Primary metric: reply rate per account. Minimum sample 250 accounts per variant, fixed before launch.
Unsubscribe rate is 2.90% on the treatment against 0.00% on control. 2.90% is above the 1.00% ceiling and the whole 95% interval (1.48–5.61%) sits above it. An unsubscribe is a permanent loss of a targetable account. Healthy cold outbound sits well under 1%; a variant that buys replies by annoying the other 99% is spending an asset we cannot rebuy. The treatment did win the primary metric (+16.3 pp on reply), and that is exactly the trade we are refusing: the lift is real and it is being paid for out of the sending domain. Also watch spam complaint rate (0.72%).
sha256('exp-provocative-v1:' + account_id) mod 10000, split by variant weightPre-registered minimum: 250 per variant
States the topic: '<Account>: agent reliability'.
Pipeline $1.4M
Implies a problem the reader has to open the email to resolve, with no value claim.
Pipeline $1.1M
Per-metric results · rates with 95% Wilson intervals on a shared scale
| Metric | Control | Treatment | Rate with 95% CI | Abs. lift (95% CI) | Rel. lift | p-value | Verdict |
|---|---|---|---|---|---|---|---|
Reply ratePrimary MDE +10.6 pp · acts above +2.0 pp | 20.7% 55 / 266 | 37.0% 102 / 276 | +16.3 pp +8.7 pp to +23.6 pp | +79% | < 0.001 | Treatment wins Treatment beats control by +16.3 pp (95% CI +8.7 to +23.6 pp, p = 0.000). Reject H0. The whole difference interval clears the 2.0 pp bar, so the lift is worth the switching cost. | |
Positive reply rate MDE +8.8 pp · acts above +1.5 pp | 11.3% 30 / 266 | 10.5% 29 / 276 | −0.8 pp −6.1 pp to +4.5 pp | −7% | 0.773 | No significant difference p = 0.773; the 95% CI for the difference (-6.1 to +4.5 pp) includes zero. We cannot reject H0. | |
Meeting rate MDE +7.4 pp · acts above +1.0 pp | 6.8% 18 / 266 | 6.2% 17 / 276 | −0.6 pp −4.9 pp to +3.6 pp | −9% | 0.774 | No significant difference p = 0.774; the 95% CI for the difference (-4.9 to +3.6 pp) includes zero. We cannot reject H0. | |
Opportunity rate MDE +6.7 pp · acts above +0.5 pp | 4.9% 13 / 266 | 4.0% 11 / 276 | −0.9 pp −4.6 pp to +2.7 pp | −18% | 0.610 | No significant difference p = 0.610; the 95% CI for the difference (-4.6 to +2.7 pp) includes zero. We cannot reject H0. |
Checked for harm, not for lift. A guardrail past its ceiling blocks the ship decision whatever the primary metric did.
| Guardrail | Control | Treatment | Ceiling | Treatment vs ceiling | Status |
|---|---|---|---|---|---|
Hard bounce rate Sustained hard bounces above ~5% tell mailbox providers the list is unverified and the sending domain starts getting filtered. It is a list-quality signal, so a variant should not move it at all. | 1.88% 5 / 266 | 1.09% 3 / 276 | 5.00% | Within limit 1.09%, and the 95% interval stays under the 5.00% ceiling. | |
Unsubscribe rate An unsubscribe is a permanent loss of a targetable account. Healthy cold outbound sits well under 1%; a variant that buys replies by annoying the other 99% is spending an asset we cannot rebuy. | 0.00% 0 / 266 | 2.90% 8 / 276 | 1.00% | Breached 2.90% is above the 1.00% ceiling and the whole 95% interval (1.48–5.61%) sits above it. | |
Spam complaint rate Gmail and Yahoo bulk-sender rules put the hard limit at 0.3% and the target at 0.1%. Crossing it throttles every campaign from the domain, not just this one. | 0.00% 0 / 266 | 0.72% 2 / 276 | 0.30% | Watch 0.72% is over the 0.30% ceiling, but the interval (0.20–2.60%) still reaches back under it. Not proven; not ignorable. | |
Negative reply rate Explicitly hostile or 'do not contact us' replies. They burn the account for future plays and are the part of a raised reply rate that looks like success in a dashboard and like damage in a CRM. | 2.63% 7 / 266 | 10.14% 28 / 276 | 8.00% | Breached +7.51 pp worse than control, and the 95% interval starts at +3.45 pp — past the 2.50 pp regression we tolerate. |
This is why reply rate alone is a bad objective. A subject line that implies a problem the reader has to open the email to resolve reliably earns replies — and unsubscribes, spam complaints and “take us off your list” from everyone it annoyed. The reply lands this quarter; the burnt domain lands next one, on someone else’s campaign.