Design an A/B Testing System — System Design Interview Practice

Interview prompt

Design an experimentation platform that assigns users consistently to variants, captures exposure and outcome events, detects sample-ratio or data-quality issues, and produces decision-ready analyses.

• Define experiment lifecycle, targeting, mutual exclusion, allocation, deterministic assignment, exposure logging, metrics, guardrails, and stopping rules.
• Keep assignment fast and stable across retries/devices where intended; support ramping, holdouts, overrides, and emergency kill switches.
• Separate flag evaluation from event collection and statistical analysis; make experiment definitions and analysis snapshots versioned.
• Explain interference, novelty, missing exposure, sample-ratio mismatch, privacy, sequential peeking, auditability, and degraded evaluation.

Requirements and scale assumptions

• Create experiments, define populations/variants/metrics, assign subjects, evaluate flags, record exposures/outcomes, and analyze results.
• Support gradual rollout, holdout groups, mutually exclusive layers, audience exclusions, guardrail alerts, approvals, and experiment history.
• Make assignments deterministic and auditable, handle identity changes, replay events, correct metrics, and stop unsafe variants immediately.
• Return a stable assignment with p95 evaluation under 20ms and publish trusted analyses only after exposure/data-quality checks.
• Scale to 1,000 concurrent experiments and 100M subjects without a single hot key or unbounded synchronous work.
• Do not lose committed state; make retries and duplicate events safe.
• Degrade safely when downstream workers, caches, or external dependencies fail.
• 100M subjects, 1,000 experiments, and 10M exposure events per minute
• Partition by the primary tenant, user, item, or geographic key and isolate hot partitions.
• Keep serving state bounded; retain raw events or durable records for replay and auditing.

• Peak scale: 1k experiments; 10M exposures/min — Capacity assumption that drives partitioning and backpressure.
• Latency target: assignment p95 < 20ms; stable hash — User-facing budget for the primary request or read path.
• Durable boundary: Committed before async — Versioned experiment definitions and exposure/outcome events are authoritative; analyses are derived snapshots.
• Async boundary: At-least-once workers — Keep Deterministic hash function for assignment, Feature flag system integration, Data pipeline for metrics collection off the synchronous path.

Key entities

• ExperimentDefinitionexperimentId, version, variants, eligibility, allocation, status

  Versioned a b testing system hypothesis and assignment contract.

• AssignmentsubjectId, experimentId, variant, definitionVersion, assignedAt, override

  Deterministic, sticky assignment with exposure metadata.

• ExposureEventeventId, subjectId, experimentId, variant, occurredAt, schemaVersion

  Deduplicated evidence that a subject actually saw a a b testing system variant.

• MetricSnapshotexperimentId, cohort, metric, window, estimate, guardrailStatus

  Reproducible metric and guardrail result for decision-making.

Data flow

1. 1. Version the experiment contractThe a b testing system catalog records hypothesis, variants, eligibility, allocation, metric definitions, and approval state.
2. 2. Assign consistentlyThe assignment service hashes the subject and experiment version, applies overrides, and emits exposure metadata.
3. 3. Collect real exposure and outcomesEvents are deduplicated and joined by cohort, definition version, and event time rather than assignment alone.
4. 4. Compute metrics and guardrailsWorkers calculate lift, uncertainty, sample-ratio mismatch, and safety metrics into a reproducible snapshot.
5. 5. Roll out or stop safelyOperators use evidence and guardrails to advance, pause, or roll back the a b testing system definition with an audit trail.

Deep dives and trade-offs

• Assignment correctnessHash on a stable subject key and experiment version so allocation remains sticky across devices where policy permits. Record exposure only when the subject could actually see the variant. Keep overrides, exclusion rules, and allocation changes explicit and auditable.
• Metric integrityUse event-time windows, deduplication, late-event policy, and cohort definitions that match the hypothesis. Alert on sample-ratio mismatch, missing exposure, instrumentation changes, and metric freshness. Analyze guardrails and segments alongside the primary metric.
• Safe decisionsRequire approval and lifecycle state transitions for a b testing system rollout, pause, and rollback. Keep definition, assignment, event, and analysis versions linked for replay. Do not let a dashboard with stale or incomplete data imply statistical certainty.
• Sticky assignment versus dynamic targetingPrefer sticky assignment for causal interpretation and make targeting changes a new version. Changing eligibility or allocation in place contaminates cohort analysis.
• Client versus server event collectionCollect exposure close to the rendering decision and supplement with server events for durable outcomes. A client-only pipeline loses events under blockers, offline use, or app termination.
• Fast decision versus statistical certaintyUse sequential monitoring with declared stopping rules and guardrails. Repeatedly checking noisy metrics until a desired result appears creates false confidence.