In A/B testing, one of the most common questions is: "How long should your A/B test run?" Determining test duration requires knowing your metric’s current level and variability (e.g., conversion rate or standard deviation), your Minimum Detectable Effect, and your desired statistical power.
When planning an A/B test, one critical question always pops up: "How long should we run this test?" The ideal answer involves some math, specifically calculating the minimum detectable effect (MDE)—the smallest improvement you care enough to reliably detect—and then seeing how long you'll need to run the test to find it with confidence (usually at 80% power).
But here's the practical reality: teams often don't have unlimited time or traffic. They frequently set the test duration first—maybe due to business cycles, deadlines, or limited traffic—and then figure out the smallest detectable improvement within that period. For instance, detecting a modest 5% uplift might need around 100,000 visitors per variant. Want to detect something as small as 1%? You'll need way more time and traffic. So, balancing statistical precision with real-world practicality often means accepting that smaller effects might slip through the cracks, but at least you'll confidently detect meaningful changes within your constraints.
Consider where you want to place your focus, and you’ll naturally gravitate toward one of two approaches—each with its own trade-offs:
Choosing between these isn't about right or wrong—it's about trade-offs. Teams that prioritize agility often accept larger detectable effects. Those committed to precision might run tests longer. Know your constraints, know your goals, and choose accordingly.
Smart A/B testing isn't just about running tests—it's about planning them thoughtfully in advance. Setting clear expectations around your MDE, test duration, and statistical power means you'll interpret your results with far greater confidence. Why does this matter?
Because relying solely on the famous p-value (usually p<0.05) can mislead you. A "statistically significant" result might mean very little practically, while a non-significant one could simply reflect insufficient data rather than true neutrality. Companies like Netflix and Microsoft have learned this lesson: they don't just chase significance—they weigh practical impact, statistical power, and real-world implications.
In short, proper planning ensures you don't mistake noise for signal, helping your team make smarter, more informed decisions about rolling out new features or improvements. Keep this balance in mind, and you'll find yourself running tests that drive genuine business value.
Businesses often structure tests in 1-week cycles to match natural user behavior rhythms—think weekly content updates or features that users revisit every seven days. To illustrate what's possible in a single cycle, let's lean on real industry benchmarks. In Databox's analysis of hundreds of email CTAs—also cited by HubSpot¹—over 40% of contributors see a 3–5% click-through rate, while nearly 15% exceed 10% CTR.
To confidently interpret your A/B test after 7 days—assuming users/day (3,500 per arm) and a baseline conversion rate of % CTR—your experiment is sized to detect an absolute change of 0.0 % or more.
(α = 0.05 ⇒ 5% false-positive risk; power = 80% ⇒ 20% false-negative risk for a true 0.0% lift.)
That 0.0% is your Minimum Detectable Effect (MDE): the smallest lift you can reliably spot with 80% power (i.e. you'll detect a true effect of this size in roughly 4 out of 5 tests).
In practice, these numbers aren't just window dressing—they directly shape what your test can (and can't) detect:
Without planning:
Imagine you see a 0.0% lift 3% → 0.0% after 7 days, but your p-value is 0.10 (>0.05, not statistically significant). Without having planned ahead, you might wrongly dismiss this promising lift as "no effect," unaware your test didn't have enough users to reliably detect lifts smaller than 0.0%.
With planning:
By sizing your test upfront for an MDE of 0.0% (80% power → you'd detect a true 0.0% lift in roughly 4 out of 5 tests), you immediately recognize the same 0.0% lift as inconclusive rather than "no effect." To reliably detect smaller effects like this 0.0% bump, you'd know ahead to increase your sample size.
Plan your sample size and MDE ahead so you know clearly when a lift is real, inconclusive, or simply underpowered.
Join our early-access pilot: an LLM-powered guide to setting up practical, accurate A/B tests so you can interpret results—and act on them—with real confidence. Fill out the form below to get started!