Email A/B Testing: What to Test, How to Read Results
Email A/B testing (also called split testing) is the practice of sending two or more versions of an email to different segments of your list to determine which version performs better. It is the only reliable way to learn what actually works with your specific audience — because what works in a case study about another brand's list may not work with yours.
Quick Answer: A/B test one variable at a time. Send to a statistically significant sample (at minimum 500–1,000 recipients per variant). Wait for 4–24 hours before declaring a winner. The variables that move the needle most: subject line, sender name, CTA text, and email length. Start with subject lines — they impact everything.
Why A/B Testing Matters (And Why Most Teams Do It Wrong)
The case for testing is simple: email performance is counterintuitive. Subject lines you think are clever underperform obvious ones. Long copy sometimes beats short copy. A/B testing removes opinion and replaces it with evidence.
The most common A/B testing mistakes:
- Testing too many variables at once — You can't know what caused the difference if you changed 5 things
- Calling tests early — Declaring a winner after 2 hours when you need 24 hours of data
- Testing on too small a sample — A 3% lift on 200 recipients is statistically meaningless
- Testing things that don't matter — Button color vs. button text; one moves the needle, one doesn't
- Never acting on results — Running tests and then ignoring what you learn
What to A/B Test (Prioritized by Impact)
Tier 1: Test These First (Highest Impact)
Subject line
This is the highest-leverage test in email. Subject line determines open rate, which affects everything downstream. Even a 5% lift in open rate is significant at scale.
Test:
- Long vs. short (50 characters vs. 20 characters)
- Question vs. statement
- Specific vs. vague
- Personalized ({{first_name}}) vs. not
- Emoji vs. no emoji
- Urgency vs. no urgency
Sender name
Who the email is from has a bigger impact on open rates than most marketers realize.
Test:
- Company name ("MailGlider") vs. person + company ("Alex from MailGlider")
- Founder name vs. team name
- First name only vs. full name
Send time
Day of week and time of day affect open rates, but the "best" time varies by audience.
Test:
- Tuesday vs. Thursday
- Morning (9am) vs. midday (12pm) vs. evening (6pm)
- Weekday vs. weekend (for consumer brands)
Tier 2: Test These for Click Rate Improvement
CTA text
The difference between "Get started" and "Start your free trial" can be significant.
Test:
- Generic vs. specific ("Learn more" vs. "Download the guide")
- First person vs. second person ("Start my trial" vs. "Start your trial")
- Action verb vs. outcome ("Download" vs. "Get the template")
Email length
There's no universal winner — it depends on your audience and content type.
Test:
- Short (under 100 words) vs. medium (200–400 words)
- Text-only vs. image-heavy
- Single CTA vs. multiple sections with multiple CTAs
CTA placement
Where you put the CTA affects whether people click it.
Test:
- Above the fold vs. at the end
- Text link vs. button
- One CTA vs. two (beginning and end)
Tier 3: Fine-Tuning Tests (Lower Impact, Worth Testing Eventually)
- Preview text (paired with subject line testing)
- Personalization level (no personalization vs. first name vs. behavioral)
- Image inclusion vs. text-only
- Bullet points vs. paragraphs
- Social proof placement (top vs. bottom of email)
How to Run a Valid A/B Test
Step 1: Define Your Hypothesis
A test without a hypothesis is just a guess. Before sending, write:
"I believe [change] will result in [outcome] because [reason]."
Example: "I believe removing the product image from the abandoned cart email and leading with a text testimonial will increase click rate because our audience has told us social proof is their primary purchase driver."
Step 2: Isolate One Variable
Change exactly one thing. If you change the subject line AND the CTA AND the email length, you won't know which change caused the difference.
Step 3: Determine Your Sample Size
As a rule of thumb:
- Minimum per variant: 500–1,000 contacts
- Recommended for confident results: 2,000–5,000 contacts per variant
- Small list? You can still run tests — just know that smaller samples mean lower confidence and longer run times to reach significance
Step 4: Set a Primary Metric Before Sending
Decide in advance what you're measuring:
- Subject line test → measure open rate
- CTA test → measure click rate
- Full email test → measure conversion or revenue
Changing your success metric after seeing results invalidates the test.
Step 5: Run the Test Long Enough
- For open rates: 4–8 hours is usually enough (people check email quickly)
- For click rates: 24 hours minimum; 48 hours for high confidence
- For conversions: 3–7 days (people don't always purchase immediately after opening)
Step 6: Evaluate Statistical Significance
A/B test results aren't real unless they're statistically significant. Most ESPs show a confidence level. Look for 95% confidence before acting on results.
If your ESP doesn't show statistical significance, use a free calculator — AB Testguide, Optimizely's calculator, or Neil Patel's A/B test significance tool.
Step 7: Document, Implement, and Build on Results
Every test result — win or lose — is information. Document:
- What you tested
- The hypothesis
- The result
- Whether to implement
- The next test this suggests
Build a testing log. Over 6–12 months, this becomes a proprietary understanding of your audience that compounds in value.
Reading Your Results Without Fooling Yourself
Common Interpretation Errors
Confusing correlation with causation
The variant sent on a Tuesday morning might outperform the one sent on Friday afternoon — but is that because of the email content, or the send time?
Survivorship bias in subject line tests
A subject line that gets a 45% open rate on a small warm segment will not get 45% when sent to the full list. Scale affects performance.
Celebrating small differences on small samples
A 2% lift on 100 recipients is not a meaningful result. The same 2% lift on 5,000 recipients, held at 95% confidence, is actionable.
Acting on one test as if it's a universal rule
"Short subject lines always beat long ones" is not true — it depends on your audience, industry, and email type. Test repeatedly and treat results as hypotheses, not laws.
Sample Testing Roadmap (First 6 Months)
| Month | Focus | Variable | Metric |
|---|---|---|---|
| 1 | Subject line length | Short vs. long | Open rate |
| 1 | Sender name | Company vs. person | Open rate |
| 2 | CTA text | Generic vs. specific | Click rate |
| 2 | CTA placement | Top vs. bottom | Click rate |
| 3 | Email length | Short vs. long | Click rate |
| 3 | Send time | Tuesday 9am vs. Thursday 12pm | Open rate |
| 4 | Personalization | No personalization vs. first name | Open rate |
| 4 | Image vs. text-only | Image-heavy vs. minimal | Click rate |
| 5–6 | Apply learnings, test new hypotheses from results |
Frequently Asked Questions
What should I A/B test first in email marketing?
Start with subject lines. They directly impact open rates, which affects everything downstream. Run 3–4 subject line tests before moving to body copy or CTA tests. You'll learn more about your audience's preferences in 4 tests than in a year of guessing.
How large does my email list need to be for A/B testing?
Ideally 1,000+ contacts per variant (2,000+ total for a standard A/B test). With smaller lists, you can still test, but results will have lower confidence. At under 200 contacts per variant, treat results as directional, not conclusive.
How long should I run an email A/B test?
For open rate results: 4–8 hours. For click rate: 24–48 hours. For conversion tracking: 3–7 days. Never cut a test short because one variant looks obviously better early — early results are often misleading.
What does statistical significance mean in email testing?
Statistical significance tells you how likely it is that the difference you observed is real and not due to random chance. At 95% significance, there's a 5% chance the result is random noise. Most ESPs calculate this automatically. Don't act on results below 90% confidence.
Can I run A/B tests on automated email sequences?
Yes, and you should — especially on welcome emails and abandoned cart sequences. These go to every new user, so even small improvements compound significantly. Most major ESPs (Klaviyo, ActiveCampaign, HubSpot) support automation A/B testing natively.
Conclusion
A/B testing is how email performance compounds over time. Every valid test teaches you something concrete about your audience — knowledge that improves every future email you send.
The rules: test one variable, use an adequate sample size, wait long enough for significance, document everything. Start with subject lines. Build from there.
If you're looking for inspiration on what high-performing email looks like before you design what you're testing, browse MailGlider — a library of real campaigns from top brands across industries.
