tech

How to Run A/B Tests Without Screwing Up: A Complete Guide

By Laurent Fidahoussen on December 19, 2025

A/B testing isn’t rocket science. But it’s merciless with people who cheat on statistical rules. After 10 years analyzing tracking data, here’s what I’ve learned about running tests that actually produce reliable results.

1. Calculate Your Sample Size BEFORE Starting

Never launch a test without knowing how many visitors you need. This is the most common mistake I see.

Why it matters: If your sample is too small, you won’t detect real differences. If you stop too early because you see “significance,” you’re likely seeing noise, not signal.

How to calculate it:

The formula is: n = 2 × [σ² × (Z₁₋α/₂ + Z₁₋β)²] / δ² (simplified version assuming the same variance in both groups; calculators use the exact two-proportion formula and give a slightly larger n. See the NIST e-Handbook.)

Where:

Practical example:

You need roughly 7,500–8,200 visitors per group depending on the formula (Evan Miller’s calculator: about 8,150).

Tools to use:

Critical point: The lower your expected improvement, the larger your sample size needs to be. If you’re testing subtle changes, you’ll need massive traffic.

2. Test for at Least Two Full Business Cycles

Even if you hit statistical significance after 3 days, keep going.

Why at least two full business cycles (for most sites, two full weeks)?

  1. Business cycles: Users behave differently on weekdays vs weekends. You need to capture at least 2 complete weekly cycles to smooth out these patterns.
  2. Returning buyers: people who started their journey before the test add noise to both groups; a longer test dilutes them.
  3. Traffic source variations: Different traffic sources (organic, paid, email) arrive at different times and behave differently.

Don’t run much longer than needed

Cookie loss and outside events build up over time:

Exception for high-traffic sites: If you have massive traffic and hit your sample size in 2-3 days, you still shouldn’t stop immediately. Even with sufficient sample size, you need to validate that the pattern holds across different days and user segments.

Sources confirm this:

3. Verify Your Sample Ratio Mismatch (SRM)

This is a critical check that most people skip.

What is SRM?

Sample Ratio Mismatch occurs when your actual split doesn’t match your intended split. For example:

Why it matters:

Microsoft Research found that approximately 6% of A/B tests have SRM issues. Even a small imbalance can:

How to detect SRM:

Use a Chi-squared (χ²) goodness-of-fit test:

χ² = Σ [(Observed – Expected)² / Expected]

Threshold: use a strict cut-off (Microsoft uses p < 0.0005; many tools use p < 0.01). Source: Microsoft Research: Diagnosing Sample Ratio Mismatch.

Example calculation:

Common causes of SRM:

  1. Broken randomization: Users aren’t being assigned correctly to groups
  2. Timing issues: Variations don’t start at the same time
  3. Bot filtering: Automatic systems incorrectly removing real users
  4. Redirect problems: If variation B uses a redirect that crashes or is slow
  5. Cookie/tracking issues: Users blocking or deleting cookies

Tools:

What to do if SRM is detected:

Don’t trust your results. The standard approach is:

  1. Identify the root cause
  2. Fix it
  3. Rerun the test

Your data is compromised and any conclusions you draw are potentially invalid.

4. Never Stop a Test Just Because It’s “Significant”

This is called peeking (optional stopping), a common form of p-hacking, and it’s one of the worst mistakes in A/B testing.

The problem:

If you continuously monitor your test and stop as soon as you see p < 0.05, you’re dramatically increasing your false positive rate. You might see “significance” due to random fluctuations, not real effects.

The rule:

Wait for BOTH:

  1. Your predetermined sample size
  2. Your minimum test duration (2 weeks)

Why this is hard:

Stakeholders get impatient. Business pressure builds. You see early “wins” and want to ship. Resist. The cost of implementing a change that doesn’t actually work (or worse, hurts conversion) is far higher than waiting a few more days.

Sequential testing alternative:

If you absolutely need to check results early, use proper sequential testing methods (like Sequential Probability Ratio Test) that are designed for continuous monitoring. But don’t just “peek” at standard A/B tests.

5. Aim for 95% Confidence and 80% Power Minimum

These are two different but equally important metrics.

Confidence Level (typically 95%):

It sets how often you accept a false alarm:

Statistical Power (typically 80%):

This is the probability of detecting a real effect if one exists.

The calculation:

For a conversion rate test with:

You need roughly 7,500–8,200 visitors per variant depending on the formula (Evan Miller’s calculator: about 8,150).

Common mistake:

Many people only check confidence and ignore power. This leads to “no difference” conclusions when there actually is a difference—you just didn’t have enough data to detect it.

6. Perform Sanity Checks on Invariant Metrics

Before trusting your results, verify that metrics that shouldn’t change haven’t changed.

What are invariant metrics?

Metrics that should theoretically be unaffected by your test variation.

Examples:

Why this matters:

If invariant metrics change, it’s a red flag that something is wrong with your test implementation:

What to check:

  1. No external shocks: Were there any campaigns, bugs, traffic spikes, or events during the test?
  2. Stable traffic sources: Did the mix of traffic sources remain consistent?
  3. No changes to other parts of the site: Nothing else was deployed that could affect results?
  4. Consistent user experience: Both variations load at the same speed? No errors?

Example of catching implementation bugs:

At one company, they noticed that pageviews increased in the test variation. This made no sense for a button color test. Investigation revealed that the variation was causing an extra page reload due to a JavaScript error. Without this sanity check, they would have trusted corrupted data.

7. Understand Your Test Statistics (But Keep It Simple)

You don’t need to be a statistician, but you should understand the basics.

For conversion rate tests:

The z-statistic is: z = (p₁ – p₂) / SE

Where SE (Standard Error) is: SE = √[p(1-p) × (1/n₁ + 1/n₂)]

If |z| > 1.96 → significant at 95% confidence level

Chi-squared test alternative:

χ² = Σ[(Observed – Expected)² / Expected]

Both methods work. Chi-squared is often easier for contingency tables.

Most important principle:

Statistical significance ≠ business impact

A test can be statistically significant (p < 0.05) but have such a small effect that it’s not worth implementing. Always consider:

Additional Best Practices

Avoid multiple testing problems:

If you test multiple variations or multiple metrics, adjust your significance threshold. The Bonferroni correction is conservative but simple: divide your alpha by the number of comparisons.

One change at a time:

Test one variable at a time when possible. If you change both the button color AND the copy, you won’t know which drove the change.

Document everything:

Pre-register your analysis plan:

Decide ahead of time:

This prevents post-hoc rationalization and HARKing (Hypothesizing After Results are Known).

Common Pitfalls to Avoid

  1. Testing too many things at once: Multiple concurrent tests can interact
  2. Changing the test mid-flight: Don’t modify variations or targeting during a test
  3. Trusting small samples: Even if “significant,” small samples are unreliable
  4. Ignoring seasonality: Don’t run tests that span major holidays or events
  5. Not documenting learnings: Failed tests teach you as much as winners

The Bottom Line

A/B testing isn’t complicated, but it’s unforgiving with those who cut corners on statistical rules.

The four most critical rules:

  1. Calculate sample size before starting
  2. Test for minimum 2 weeks
  3. Check for Sample Ratio Mismatch
  4. Never stop early just because it’s “significant”

Get these right, and your tests will be reliable. Skip them, and you’re just generating noise.

Statistics checked against the ASA statement on p-values and the NIST e-Handbook. The principles come from 10 years of hands-on tracking and analytics work for telecom, travel and e-commerce companies.

Sources

Not sure your own tracking is right?

Book a call Or start with the free analytics audit →