Glossary of Terms
Term | Definition |
---|---|
A/B Testing | Comparing two versions of an element (subject line, design, CTA) to determine which performs better. |
Multivariate Testing (MVT) | Testing multiple variables (e.g., subject + creative) simultaneously to see which combination performs best. |
CTR (Click-Through Rate) | Percentage of email recipients who clicked at least one link. |
CTO (Click-to-Open Rate) | Clicks divided by total opens—shows how compelling your content is post-open. |
Click-to-Order Rate | Percentage of clicks that resulted in a completed purchase. |
ABV (Average Basket Value) | The average value of orders placed via a specific email variant. |
Revenue per Email Sent (RPE) | Total revenue divided by the number of emails sent—key metric for profitability. |
ESP (Email Service Provider) | Tools like Klaviyo, Mailchimp, or Iterable used to design, send, and track email campaigns. |
Table of contents
Why Email Testing Isn’t Optional Anymore
In today’s crowded inbox, guessing doesn’t cut it.
Data-driven decisions separate mediocre campaigns from money-makers. But here’s the twist:
Testing the wrong way is as dangerous as not testing at all.
Most brands run A/B tests based solely on open or click rates—and stop there. But those metrics don’t always drive revenue.
Let’s dive into a real-world test that shows why going deeper matters, what to test (and when), and how to build a high-performing testing framework that actually moves your bottom line.
The Test Setup: A Real Client Case Study
For this test, we explored two core variables across four schedules:
Email Version | Subject Line | Creative |
---|---|---|
Schedule A | SL1 | C1 |
Schedule B | SL1 | C2 |
Schedule C | SL2 | C1 |
Schedule D | SL2 | C2 |
All emails were sent at the same time, and tracked across:
- Opens
- Clicks
- Click-to-Order (CTO)
- Revenue per email
- ABV (Average Basket Value)
The Results (What Looked Good… and What Actually Was)
Subject Line Performance
- SL1 had higher open rates
- SL2, despite lower opens, generated higher revenue per send
Creative Performance
- C1 had better CTRs
- C2, though weaker on clicks, delivered higher Click-to-Order conversions and larger average baskets
The Hidden Danger: Relying on Automated Testing Tools

If we had let an ESP (like Klaviyo or Mailchimp) decide the “winner” based on opens or clicks alone?
They would’ve chosen SL1 and C1.
That would’ve cost our client real revenue.
Why?
Because SL2 and C2, despite lower top-funnel engagement, converted better and drove higher order value.
Takeaway: The metrics most tools optimize for (opens, clicks) often don’t correlate with profit.
What Most Marketers Get Wrong About Email Testing
Misstep | Why It Hurts |
---|---|
Testing too many variables at once | You can’t isolate what worked |
Choosing winners based only on open/click rates | You might kill revenue |
Relying 100% on automation | ESPs rarely optimize for actual sales |
Not defining the success metric before testing | Leads to biased interpretation or false positives |
Best Practices for Smarter, Revenue-Driven Email Testing
1. Always Define the Goal First
Every test must start with one question:
“What are we optimizing for?”
That could be:
- Opens (when testing subject lines for awareness)
- Clicks (when measuring engagement)
- Orders or RPE (when focused on revenue)
Examples of Test Goals by Campaign Type:
Campaign Type | Primary Metric |
---|---|
Welcome Series | Click-to-Open Rate |
Promo Blast | Revenue per Email |
Product Launch | CTR + Click-to-Order |
Post-Purchase Flow | ABV or Repeat Orders |
2. Test One Variable at a Time (Whenever Possible)
Multivariate testing is useful—but complicated.
If you test subject line and creative at the same time, you won’t know what made the difference.
Start simple:
- A/B Test subject lines first
- Then test creative (hero image, CTA copy, layout)
- Then test send time or list segment
Suggested Testing Sequence:
Stage | Variable | Purpose |
---|---|---|
1 | Subject Line | Boost opens |
2 | CTA Copy | Improve click intent |
3 | Layout or Image | Refine UX |
4 | Send Time | Maximize visibility |
5 | Segment Targeting | Match content with behavior |
3. Go Beyond Clicks — Track End-to-End Metrics
Clicks are just the start. Always track downstream metrics like:
- Click-to-Order conversion
- Revenue per click
- ABV
- Unsub rate (especially on aggressive CTAs or promos)
Your test winner should always be the most profitable variant, not just the flashiest.
Test Result Comparison Table:
Metric | SL1 + C1 | SL2 + C2 |
---|---|---|
Open Rate | 28% | 22% |
CTR | 5.5% | 3.8% |
Click-to-Order | 8.2% | 14.9% |
ABV | $78 | $95 |
Revenue per Send | $1.08 | $1.62 ✅ Winner |
4. Use Automation—But Don’t Blindly Trust It
Automation is a tool. Not a decision-maker.
Use ESP tools to:
- Split sends by audience
- Randomize test groups
- Aggregate performance data
Don’t use them to:
- Auto-select winners based on partial data
- End tests early (wait at least 24–48 hrs)
- Assume “highest opens = best email”
Pro Tip: Always manually export and analyze UTM + order data from GA4 or Shopify.
5. Small Details Matter — Test These Underrated Elements
Element | Why Test It? | Example Variants |
---|---|---|
CTA Copy | Impacts click intent | “Buy Now” vs. “Explore More” |
Button Color | Affects visual hierarchy | Blue vs. Black |
Product Grid Layout | Changes perceived value | 1-column vs. 2-column |
Preheader Text | Drives opens | “Don’t miss this deal” vs. “A surprise inside” |
Personalization | Increases relevance | “Hey [First Name]” vs. generic |
6. Set Your Sample Size (and Know When You Have a Winner)
Don’t base decisions on tiny audiences.
Minimum thresholds for reliable tests:
- 1,000 recipients per variant
- 100+ opens per group (for subject line)
- 50+ clicks per group (for CTA/creative)
Use calculators like:
- ABTestGuide.com
- Evan Miller’s A/B Testing Tool
Frequently Asked Questions (FAQ)
1. What’s the most important metric to track in email testing?
It depends on your goal. For revenue-driving campaigns, prioritize Revenue per Email Sent or Click-to-Order Rate over open rate or CTR.
2. How long should I run an A/B test?
Minimum 24–48 hours. Wait until both variants have statistically significant opens/clicks before choosing a winner.
3. Can I test multiple variables at once?
Yes, through multivariate testing (MVT). But only do this if you have a large list—ideally 10,000+ subscribers per variant—to ensure reliable data.
4. What’s a good open rate for ecommerce emails?
Industry average is 15–25%. High-performing flows like Welcome or Abandoned Cart can hit 30%+.
5. Do emojis in subject lines improve performance?
Sometimes. They can boost visibility but may reduce performance in some industries. Test them—don’t assume.
6. Should I trust Klaviyo’s “winning version” in a test?
Not blindly. Klaviyo optimizes for opens/clicks—not revenue. Always cross-check performance with GA4 or your ecommerce platform.
Final Takeaway: Test With Purpose. Win With Profit.
Email testing isn’t about running random experiments.
It’s about systematically improving:
- Relevance
- Timing
- Design
- Profit
You don’t just want more clicks—you want more customers.
So define your goal. Test one variable. Track what matters. Then scale what works.
Ready to Turn Your Email Program Into a Revenue Engine?
At Blossom Ecom, we build testing strategies that don’t just generate opens—they generate cash.
Let’s review your current tests and find the revenue leaks you’re missing.
Claim your Free Email Testing Audit today—and get actionable recommendations you can implement in 48 hours.
Need help implementing this?
Let us take the hassle of managing your email marketing channel off your hands. Book a strategy call with our team today and see how we can scale your revenue, customer retention, and lifetime value with tailored strategies. Click here to get started.
Curious about how your Klaviyo is performing?
We’ll audit your account for free. Discover hidden opportunities to boost your revenue, and find out what you’re doing right and what could be done better. Click here to claim your free Klaviyo audit.
Want to see how we’ve helped brands just like yours scale?
Check out our case studies and see the impact for yourself. Click here to explore.
Read Our Other Blogs

Attract More Customers: Which Videos to Use in Your Email Marketing and Why



8 eCommerce Customer Service Mistakes You NEED to Stop Making (Like, Yesterday)



Creating The Perfect E-Commerce Tech Stack




Not Sure Where to Start?
Let's find the biggest retention opportunities in your business. Get a free Klaviyo audit or retention consultation.

