- You have traffic, a hypothesis, and a primary metric
- The change is small enough to isolate
- You do not have the volume. Use interviews or a survey
- You do not know if people can do the task at all. Usability first
- You are launching a whole product. That is a launch, maybe a beta
Steps
Hypothesis, design, and sample size are the paper. Implement and monitor are the run. Analysis and the decision sit together.
Hypothesis and design
If we change X, metric Y moves because Z. One change. Primary metric, guardrails, how long, how many people. Write it down before anyone builds.
- Product
A/B Test Hypothesis Generator
Create testable A/B test hypotheses with success metrics, variants, and implementation plans.
ab-testingexperimentationmetrics - Research
A/B Test Design Template
Design rigorous A/B tests with clear hypotheses, metrics, and success criteria.
ab-testingexperimentationproduct-analytics - Research
A/B Test Sample Size Guide
Calculate required sample size and test duration for statistically valid experiments.
ab-testingstatisticssample-size
Run
Build, QA tracking, launch. Watch sample ratio and guardrails. Do not peek at the primary metric to “see if we can stop.”
Decide
Significance is not the same as a change worth shipping. Effect size, guardrails, then ship, iterate, or drop. Write it down either way.
Frequently asked questions
Can we stop when it looks significant?
No. Wait for the sample you planned, unless the product is on fire.
We do not have the traffic. Still run this?
No. Use interviews or a survey. An underpowered test will give you a number you cannot defend.
People cannot finish the task. A/B it?
No. Run usability first. Shipping two broken variants measures which broken one loses slower.