Illustrative setup

Suppose a product is testing a new registration entry. This is a hypothetical work-note scenario, not a real project. Traffic is assigned 50/50, the test is planned for 14 days to cover 2 full weekly cycles, and a 24-hour A/A observation happens before launch.

Write the decision before the measure

State the product decision the test may support: keep the new entry, revise it, or restore the previous version. The primary measure should answer that decision; guardrails observe whether errors, latency, or downstream steps deteriorate.

If the team cannot say what a result would change, even a statistically clear outcome may have no defined use.

Confirm each item before launch

  1. Randomisation unitChoose user, device, or session, and document cross-device and anonymous-to-signed-in handling.
  2. Mutual exclusionCheck whether concurrent tests alter the same page, population, or primary measure.
  3. Measure definitionFreeze numerator, denominator, deduplication, window, time zone, and outlier rules.
  4. Sample and durationEstimate the sample from the detectable effect and baseline; do not stop for ordinary mid-test movement.
  5. Quality monitoringPrepare daily checks for allocation, missing events, version coverage, and error rate.

What the A/A observation checks

An A/A observation does not prove that groups will always match. It checks random assignment, event firing, log delay, and report calculations. A clear allocation anomaly within 24 hours should trigger implementation review before the formal test.

Write the limits of interpretation in advance

Before launch, I like to leave room in the experiment document for three outcomes: the primary measure rises, stays flat, or falls. Each outcome should point to a possible product decision and a list of evidence still missing. It is deliberately plain work, but it makes post-test discussions much less vulnerable to a convenient story.

The same page should keep the primary measure beside its guardrails. An improvement paired with a higher error rate is an incomplete result. A flat primary measure also deserves a check of assignment, event firing, and exposure before anyone decides the change had no effect.

  • The product decision attached to each outcome.
  • Claims that remain observations rather than explanations.
  • Guardrail movements that would block a release.
  • Data-quality failures that would invalidate the readout.

Set stopping rules before seeing results

If the plan says 14 days, define which safety issues allow an early stop and which ordinary fluctuations do not. Repeatedly checking the result and stopping at its most favorable point changes the false-positive risk.