Experiment analytics

Experiment analytics lives in the Marketing area of the dashboard, under Optimization.

At a glance#

What it does#

Experiment analytics is the results side of experimentation: per-variant performance for a selected experiment, with the KPIs and the breakdown by variant.

Setup and lifecycle live on A/B tests and Multivariate. This page is where you read the outcome.

When to use it#

At the end of an experiment's planned duration. Not before — reading results while a test runs, and reacting to them, is the single most common way experimentation produces wrong answers, because significance thresholds assume you decided when to stop before you started.

Workflow#

1. 1

   Check the sample size per variant first

   An underpowered test produces a difference that means nothing. If the sample is below what the MDE calculator specified, the result is not readable.

2. 2

   Read the effect size, not just the direction

   A statistically detectable 0.3% improvement may not be worth the complexity of shipping it. Significance and importance are different questions.

3. 3

   Confirm variant allocation was even

   A large imbalance between arms suggests an assignment problem, and a broken assignment invalidates the comparison.

4. 4

   Accept a null result

   "No difference" is a real, valuable answer that saves you from shipping complexity for nothing. Re-cutting the data by segment until something looks significant is how you find noise.

Permissions and prerequisites#

Requires a running or completed experiment with variant assignment happening in the app, and the metric event tracked.

Limits and edge cases#

• Per-variant metrics inherit their own caveats — client-reported revenue, attribution coverage, and so on.

• Segment slicing multiplies false positives. Every additional cut is another chance for noise to look real.

• A user counted in both arms breaks the comparison. Check allocation stability if arms look contaminated.

• Late-arriving events shift results slightly after a test ends.

Troubleshooting#

No data for an experiment. Variant assignment is not reaching the SDK, or the metric event is not arriving.

Uneven variant sizes. Assignment is not respecting the configured allocation, or a variant is being skipped by a condition in your code.

Result changed after the test ended. Late-arriving events. Expected, and usually small.

Where the data comes from#

Served by

• Experimentation

Related#

• A/B tests

• Multivariate

• Flags & config

• Marketing overview