What Is A/B Testing? | Inbound Marketing Glossary

A/B testing is an experimentation technique in which two versions (A and B) of a marketing asset—such as a web page, email, or ad—are simultaneously shown to comparable user groups under identical conditions. Key metrics like conversion rate and click-through rate are then compared to select the superior version based on data.

Why It Matters

While intuition and experience can serve as starting points for marketing decisions, failing to validate conclusions with data often leads to wasted spend. A/B testing eliminates subjective judgment by providing evidence grounded in actual user behavior. In a well-known example, the 2008 Obama presidential campaign ran approximately 500 A/B tests, boosting donation conversion rates by 49% and email sign-up rates by 161%. Because even a small change can produce a dramatic difference in conversion rates, A/B testing is a cornerstone of Conversion Rate Optimization (CRO) in inbound marketing.

How to Design an A/B Test

Formulate a hypothesis: State a specific, measurable hypothesis such as "Changing the CTA button color from blue to orange will increase click-through rate by at least 10%."
Select a primary metric: Choose one key metric—conversion rate, click-through rate, bounce rate, etc. Using multiple primary metrics makes results ambiguous.
Calculate sample size: Before launching the test, determine the required sample size. Calculations are typically based on a 95% confidence level, 80% statistical power, and the minimum detectable effect (MDE) you want to identify. For example, if the current conversion rate is 5% and you need 95% significance, each group requires roughly 6,900 or more participants.
Run the test: Randomly split traffic 50:50 and run the experiment for at least 2–6 weeks. Tests shorter than one week fail to account for day-of-week traffic fluctuations, reducing reliability.
Analyze and apply results: Once statistical significance is confirmed (p-value < 0.05), roll out the winning version to all users.

Testable Elements

Headlines and copy: Changing a single headline can shift click-through rates by over 20%.
CTA (Call-to-Action): Experiment with button text ("Free Trial" vs. "Get Started Now"), color, position, and size.
Landing page layout: Compare the presence or absence of a hero image, the number of form fields, and the placement of social proof (testimonials, logos).
Email: Test subject lines, sender names, body length, and send times.
Pricing and offers: Discount presentation format (flat amount vs. percentage), bundle configurations, and similar variables are all meaningful test candidates.

Common Mistakes

Early peeking: Stopping a test prematurely because early numbers look promising can mistake random fluctuation for a real effect. Follow the "no peeking" principle and avoid checking results for at least seven days.
Changing multiple variables at once: If you alter the headline and CTA simultaneously, you cannot determine which element drove the performance difference. Change only one variable at a time. To test multiple variables simultaneously, design a separate multivariate test.
Insufficient sample size: Running a test with too little traffic makes it impossible to achieve statistical significance. Use a sample size calculator beforehand to confirm the minimum traffic required before launching.
Over-generalizing results: Applying results obtained during a specific season or promotional period year-round can create a gap between expected and actual performance. Always verify that the test environment matches the deployment environment.
Ignoring Type I errors: A significance level of 0.05 means there is a 5% probability the result is due to chance. If you run 20 tests, one is likely to produce a false positive. For critical decisions, cross-validate through replication.

Sources: