A/B Testing Guide: How Long Should Your Tests Run?

How Long Should Your A/B Test Run?

When planning an A/B test, one critical question always pops up: "How long should we run this test?" The ideal answer involves some math, specifically calculating the minimum detectable effect (MDE)—the smallest improvement you care enough to reliably detect—and then seeing how long you'll need to run the test to find it with confidence (usually at 80% power).

But here's the practical reality: teams often don't have unlimited time or traffic. They frequently set the test duration first—maybe due to business cycles, deadlines, or limited traffic—and then figure out the smallest detectable improvement within that period. For instance, detecting a modest 5% uplift might need around 100,000 visitors per variant. Want to detect something as small as 1%? You'll need way more time and traffic. So, balancing statistical precision with real-world practicality often means accepting that smaller effects might slip through the cracks, but at least you'll confidently detect meaningful changes within your constraints.

MDE-first or Timeframe-first: What's Your Play?

Two major approaches dominate A/B test planning, each with its own set of trade-offs:

• MDE-first: You first ask, "What's the smallest change worth noticing?" and set your sample size or duration accordingly. This ensures high statistical rigor—you won't miss subtle but valuable changes. But here's the catch: if the change you're after is small, you might end up running the test for weeks or even months, which can slow your team down.
• Timeframe-first: You decide upfront how long the test can practically run (say, two to four weeks) and then determine the smallest detectable effect within that fixed window. This method keeps tests quick and aligned with business needs, but you might miss detecting smaller, yet meaningful improvements.

Choosing between these isn't about right or wrong—it's about trade-offs. Teams that prioritize agility often accept larger detectable effects. Those committed to precision might run tests longer. Know your constraints, know your goals, and choose accordingly.

Why Planning Ahead Makes All the Difference

Smart A/B testing isn't just about running tests—it's about planning them thoughtfully in advance. Setting clear expectations around your MDE, test duration, and statistical power means you'll interpret your results with far greater confidence. Why does this matter?

Because relying solely on the famous p-value (usually p<0.05) can mislead you. A "statistically significant" result might mean very little practically, while a non-significant one could simply reflect insufficient data rather than true neutrality. Companies like Netflix and Microsoft have learned this lesson: they don't just chase significance—they weigh practical impact, statistical power, and real-world implications.

In short, proper planning ensures you don't mistake noise for signal, helping your team make smarter, more informed decisions about rolling out new features or improvements. Keep this balance in mind, and you'll find yourself running tests that drive genuine business value.