Primary metric: reply rate per account. Minimum sample 250 accounts per variant, fixed before launch.
Only 37 units in the smaller arm; the pre-registered minimum is 250. No winner is declared. No guardrail has been breached, so the test can continue. Any lift here is below the 2.0 pp we treat as worth the operational cost of changing the play.
Keep the test running. Observed differences at this sample size are mostly noise, so the dashboard declares no winner.
sha256('exp-subject-v1:' + account_id) mod 10000, split by variant weightPre-registered minimum: 250 per variant
Pipeline $249K
Pipeline $0
Smallest arm has 37 of 250 required accounts.
Per-metric results · rates with 95% Wilson intervals on a shared scale
| Metric | Control | Treatment | Rate with 95% CI | Abs. lift (95% CI) | Rel. lift | p-value | Verdict |
|---|---|---|---|---|---|---|---|
Reply ratePrimary MDE +32.0 pp · acts above +2.0 pp | 32.4% 12 / 37 | 13.0% 6 / 46 | −19.4 pp −36.9 pp to −1.4 pp | −60% | 0.033 | Insufficient sample Only 37 units in the smaller arm; the pre-registered minimum is 250. No winner is declared. Any lift here is below the 2.0 pp we treat as worth the operational cost of changing the play. | |
Positive reply rate MDE +31.0 pp · acts above +1.5 pp | 21.6% 8 / 37 | 8.7% 4 / 46 | −12.9 pp −29.4 pp to +2.6 pp | −60% | 0.096 | Insufficient sample Only 37 units in the smaller arm; the pre-registered minimum is 250. No winner is declared. | |
Meeting rate MDE +24.4 pp · acts above +1.0 pp | 5.4% 2 / 37 | 4.3% 2 / 46 | −1.1 pp −13.8 pp to +9.9 pp | −20% | 0.823 | Insufficient sample Only 37 units in the smaller arm; the pre-registered minimum is 250. No winner is declared. | |
Opportunity rate MDE +24.4 pp · acts above +0.5 pp | 5.4% 2 / 37 | 0.0% 0 / 46 | −5.4 pp −17.7 pp to +3.2 pp | −100% | 0.110 | Insufficient sample Only 37 units in the smaller arm; the pre-registered minimum is 250. No winner is declared. |
Checked for harm, not for lift. A guardrail past its ceiling blocks the ship decision whatever the primary metric did.
| Guardrail | Control | Treatment | Ceiling | Treatment vs ceiling | Status |
|---|---|---|---|---|---|
Hard bounce rate Sustained hard bounces above ~5% tell mailbox providers the list is unverified and the sending domain starts getting filtered. It is a list-quality signal, so a variant should not move it at all. | 8.11% 3 / 37 | 0.00% 0 / 46 | 5.00% | Within limit 0.00% is under the 5.00% ceiling, but with 46 units the interval still reaches 7.71% — under the limit, not yet cleared of it. Control is over the ceiling too (8.11%), so this is a programme issue, not a variant one. | |
Unsubscribe rate An unsubscribe is a permanent loss of a targetable account. Healthy cold outbound sits well under 1%; a variant that buys replies by annoying the other 99% is spending an asset we cannot rebuy. | 5.41% 2 / 37 | 0.00% 0 / 46 | 1.00% | Within limit 0.00% is under the 1.00% ceiling, but with 46 units the interval still reaches 7.71% — under the limit, not yet cleared of it. Control is over the ceiling too (5.41%), so this is a programme issue, not a variant one. | |
Spam complaint rate Gmail and Yahoo bulk-sender rules put the hard limit at 0.3% and the target at 0.1%. Crossing it throttles every campaign from the domain, not just this one. | 0.00% 0 / 37 | 0.00% 0 / 46 | 0.30% | Within limit 0.00% is under the 0.30% ceiling, but with 46 units the interval still reaches 7.71% — under the limit, not yet cleared of it. | |
Negative reply rate Explicitly hostile or 'do not contact us' replies. They burn the account for future plays and are the part of a raised reply rate that looks like success in a dashboard and like damage in a CRM. | 2.70% 1 / 37 | 4.35% 2 / 46 | 8.00% | Within limit 4.35% is under the 8.00% ceiling, but with 46 units the interval still reaches 14.53% — under the limit, not yet cleared of it. |
This is why reply rate alone is a bad objective. A subject line that implies a problem the reader has to open the email to resolve reliably earns replies — and unsubscribes, spam complaints and “take us off your list” from everyone it annoyed. The reply lands this quarter; the burnt domain lands next one, on someone else’s campaign.