A/B tests
A/B tests in the Marketing area (Optimization) — route /marketing/ab-tests.
A/B tests lives in the Marketing area of the dashboard, under Optimization.
At a glance#
| Dashboard route | /marketing/ab-tests |
| Area | Marketing (marketing) |
| Group | Optimization |
| Platforms | Available for every app platform. |
What it does#
A/B tests: create an experiment with variants and allocations, start and stop it, and — importantly — a minimum detectable effect calculator for working out how much traffic you need before you start.
Results are read on Experiment analytics. This page is the setup and lifecycle.
When to use it#
When a decision is genuinely uncertain and the outcome is measurable. That combination is rarer than it sounds — most product decisions are either obvious or not measurable at your traffic level, and testing those wastes weeks to learn nothing.
The MDE calculator exists to tell you that before you commit. Use it first, every time.
Workflow#
Run the MDE calculator before creating the test
It tells you the smallest effect your traffic can detect. If that number is larger than the effect you expect, the test cannot succeed and should not be run.
Decide the metric and the duration in advance
Write both down before starting. Choosing the metric after seeing the data is how a null result becomes a false positive.
Use two variants
Every extra variant splits your traffic and raises the false-positive rate. Two-way tests reach a conclusion; four-way tests usually do not. For genuinely multi-factor questions use Multivariate and expect to need much more traffic.
Run to the planned end, then stop
Stopping when a result looks good is the most common way A/B testing produces confident wrong answers.
Permissions and prerequisites#
Requires variant assignment in your app — the SDK deciding which variant a user gets — and the metric event being tracked.
Limits and edge cases#
A test needs traffic proportional to the subtlety of the effect. Small effects need very large samples; the calculator quantifies this.
Variant allocation must be stable per user. A user seeing both variants pollutes both arms.
Novelty effects distort short tests. A new thing gets attention for being new.
A stopped test's assignment stops too. Decide what users see afterwards.
Troubleshooting#
Test running with no data. Variant assignment is not happening in the app, or the metric event is not arriving.
Results flip back and forth. Insufficient sample. This is exactly what the MDE calculator warns about.
Both variants perform identically. A real and useful result: the thing you changed does not matter. Ship the simpler one.
Where the data comes from#
From the SDK
Served by
Experimentation