Primary metric: positive reply rate per account. Minimum sample 120 accounts per variant, fixed before launch.
p = 0.129; the 95% CI for the difference (-1.9 to +15.1 pp) includes zero. We cannot reject H0. This test could only have detected about 13.3 pp, well above the 1.5 pp bar we would act on — the null result is underpowered, not evidence of no effect. Keeping the current play costs nothing; switching would.
sha256('exp-funding-v1:' + account_id) mod 10000, split by variant weightPre-registered minimum: 120 per variant
Persona pain + value prop; no trigger reference.
Pipeline $755K
Opens with the verified funding round and its implication.
Pipeline $356K
Per-metric results · rates with 95% Wilson intervals on a shared scale
| Metric | Control | Treatment | Rate with 95% CI | Abs. lift (95% CI) | Rel. lift | p-value | Verdict |
|---|---|---|---|---|---|---|---|
Reply rate MDE +14.9 pp · acts above +2.0 pp | 21.3% 34 / 160 | 37.8% 54 / 143 | +16.5 pp +6.3 pp to +26.4 pp | +78% | 0.002 | Treatment wins Treatment beats control by +16.5 pp (95% CI +6.3 to +26.4 pp, p = 0.002). Reject H0. | |
Positive reply ratePrimary MDE +13.3 pp · acts above +1.5 pp | 13.8% 22 / 160 | 20.3% 29 / 143 | +6.5 pp −1.9 pp to +15.1 pp | +47% | 0.129 | No significant difference p = 0.129; the 95% CI for the difference (-1.9 to +15.1 pp) includes zero. We cannot reject H0. This test could only have detected about 13.3 pp, well above the 1.5 pp bar we would act on — the null result is underpowered, not evidence of no effect. | |
Meeting rate MDE +12.3 pp · acts above +1.0 pp | 10.6% 17 / 160 | 8.4% 12 / 143 | −2.2 pp −9.0 pp to +4.7 pp | −21% | 0.509 | No significant difference p = 0.509; the 95% CI for the difference (-9.0 to +4.7 pp) includes zero. We cannot reject H0. | |
Opportunity rate MDE +10.2 pp · acts above +0.5 pp | 5.6% 9 / 160 | 4.9% 7 / 143 | −0.7 pp −6.1 pp to +4.8 pp | −13% | 0.777 | No significant difference p = 0.777; the 95% CI for the difference (-6.1 to +4.8 pp) includes zero. We cannot reject H0. |
Checked for harm, not for lift. A guardrail past its ceiling blocks the ship decision whatever the primary metric did.
| Guardrail | Control | Treatment | Ceiling | Treatment vs ceiling | Status |
|---|---|---|---|---|---|
Hard bounce rate Sustained hard bounces above ~5% tell mailbox providers the list is unverified and the sending domain starts getting filtered. It is a list-quality signal, so a variant should not move it at all. | 1.25% 2 / 160 | 2.10% 3 / 143 | 5.00% | Within limit 2.10% is under the 5.00% ceiling, but with 143 units the interval still reaches 5.99% — under the limit, not yet cleared of it. | |
Unsubscribe rate An unsubscribe is a permanent loss of a targetable account. Healthy cold outbound sits well under 1%; a variant that buys replies by annoying the other 99% is spending an asset we cannot rebuy. | 0.63% 1 / 160 | 0.70% 1 / 143 | 1.00% | Within limit 0.70% is under the 1.00% ceiling, but with 143 units the interval still reaches 3.85% — under the limit, not yet cleared of it. | |
Spam complaint rate Gmail and Yahoo bulk-sender rules put the hard limit at 0.3% and the target at 0.1%. Crossing it throttles every campaign from the domain, not just this one. | 0.63% 1 / 160 | 0.00% 0 / 143 | 0.30% | Within limit 0.00% is under the 0.30% ceiling, but with 143 units the interval still reaches 2.62% — under the limit, not yet cleared of it. Control is over the ceiling too (0.62%), so this is a programme issue, not a variant one. | |
Negative reply rate Explicitly hostile or 'do not contact us' replies. They burn the account for future plays and are the part of a raised reply rate that looks like success in a dashboard and like damage in a CRM. | 2.50% 4 / 160 | 4.90% 7 / 143 | 8.00% | Within limit 4.90% is under the 8.00% ceiling, but with 143 units the interval still reaches 9.76% — under the limit, not yet cleared of it. |
This is why reply rate alone is a bad objective. A subject line that implies a problem the reader has to open the email to resolve reliably earns replies — and unsubscribes, spam complaints and “take us off your list” from everyone it annoyed. The reply lands this quarter; the burnt domain lands next one, on someone else’s campaign.