Multivariate tests
Multivariate tests in the Marketing area (Optimization) — route /marketing/multivariate.
Multivariate tests lives in the Marketing area of the dashboard, under Optimization.
At a glance#
| Dashboard route | /marketing/multivariate |
| Area | Marketing (marketing) |
| Group | Optimization |
| Platforms | Available for every app platform. |
What it does#
Multivariate testing: experiments with several variants, each with its own allocation percentage, started and stopped from here. Results are read on Experiment analytics.
Where an A/B test compares two options, this compares many — which lets you test combinations rather than one change at a time.
When to use it#
Rarely, and only at real traffic volume. The honest guidance for most teams is: use A/B tests instead.
The reason is arithmetic. Every variant you add splits your traffic and raises the chance that one arm looks like a winner through noise alone. A four-variant test needs substantially more than twice the traffic of a two-variant one to reach the same confidence — and if you have that much traffic, you could have run two sequential A/B tests and learned more.
Workflow#
Justify each variant
If you cannot say what you will learn from a variant, remove it. Every one costs statistical power.
Set allocations deliberately
Uneven allocation is legitimate — a smaller share for a risky variant — but it means that arm reaches significance later.
Plan for a longer run than you want
Multivariate tests take considerably more time. Decide the duration up front and hold to it.
Read results with the multiple-comparison problem in mind
The best-looking arm of six is the best-looking arm of six. That is not the same as being better.
Permissions and prerequisites#
Requires variant assignment in the app respecting the configured allocations, and the metric event tracked.
Limits and edge cases#
Underpowered arms are the normal failure mode, not the exception.
Allocations must sum sensibly and assignment must be stable per user.
Interaction effects are what this is for — testing two factors together — and they are also the hardest thing to read from the results.
A stopped test stops assigning. Decide what users see afterwards.
Troubleshooting#
Every arm looks similar and none is significant. Almost always insufficient per-variant sample. Fewer variants, or more time.
One arm has far fewer users than allocated. Assignment is not honouring the allocation, or a condition in your code is skipping it.
A winner that does not hold up when shipped. The classic multiple-comparison false positive. Re-test the winner against control as a two-way A/B.
Where the data comes from#
From the SDK
Served by
Experimentation