Design a Real-time Model Monitoring and Drift Detection System — System Design Interview Practice

Interview prompt

Design a production ML monitoring platform that detects feature drift, prediction skew, data quality failures, and delayed model-performance degradation, then gates or triggers retraining safely.

• Define telemetry, feature/prediction/label windows, reference baselines, statistical tests, minimum samples, and model-version isolation.
• Distinguish data quality, training-serving skew, covariate drift, concept drift, and delayed-label performance with actionable thresholds.
• Separate ingestion from dashboard queries and alert evaluation; make monitors replayable, versioned, and idempotent.
• Explain alert deduplication, privacy and sampling, missing labels, baseline changes, rollback gates, retention, and degraded operation.

Requirements and scale assumptions

• Ingest feature, prediction, outcome, model-version, and data-quality telemetry; compute windowed statistics and drift/performance measures.
• Expose dashboards and APIs for model, feature, segment, region, and time-window drill-down with sample counts, freshness, and confidence.
• Create deduplicated alerts, route them to owners, trigger approval-aware retraining or rollback workflows, and replay historical data.
• Detect distribution and quality issues within 15 minutes and compute delayed-label performance as labels arrive.
• Scale to millions of predictions per hour and thousands of model versions without a single hot key or unbounded synchronous work.
• Do not lose committed state; make retries and duplicate events safe.
• Degrade safely when downstream workers, caches, or external dependencies fail.
• Millions of prediction records per hour across thousands of model versions
• Partition by tenant, model version, feature, time window, and cohort; pre-aggregate for interactive queries.
• Keep serving state bounded; retain raw events or durable records for replay and auditing.

• Peak scale: 10M predictions/hour; 2k model versions — Capacity assumption that drives partitioning and backpressure.
• Latency target: p95 detection < 15 minutes — User-facing budget for the primary request or read path.
• Durable boundary: Committed before async — Immutable telemetry and monitor definitions are authoritative; aggregates can be recomputed.
• Async boundary: At-least-once workers — Keep Use KS-test, PSI, or Jensen-Shannon divergence for drift, Implement reference-window vs sliding-window comparison, Use Evidently AI or Whylogs for monitoring off the synchronous path.

Key entities

• DatasetVersiondatasetId, version, schemaHash, qualityStatus, lineage, createdAt

  Immutable real time model monitoring and drift detection system input version used for reproducible training, evaluation, or replay.

• FeatureSnapshotentityId, featureSetVersion, eventTime, values, sourceWatermarks

  Point-in-time real time model monitoring and drift detection system features with source watermarks so online and offline values can be compared.

• TrainingRunrunId, datasetVersion, codeVersion, metrics, artifactUri, status

  Audited real time model monitoring and drift detection system run that records data, code, dependency, and evaluation lineage.

• ModelVersionmodelId, version, stage, schema, qualityGates, endpoint

  A promotable real time model monitoring and drift detection system model version with rollout state, contract, and rollback metadata.

Data flow

1. 1. Register and validate training dataThe real time model monitoring and drift detection system gateway records an immutable dataset version, schema, lineage, quality status, and privacy disposition.
2. 2. Build point-in-time featuresFeature workers join real time model monitoring and drift detection system inputs using event-time watermarks, prevent leakage, and publish the same feature contract for training and serving.
3. 3. Train and evaluate asynchronouslyThe orchestrator schedules real time model monitoring and drift detection system runs with checkpointed artifacts, reproducible environments, and metrics tied to the exact input versions.
4. 4. Gate and serve a model versionA registry compares real time model monitoring and drift detection system quality, bias, safety, and compatibility gates before canary or production rollout with an immediate rollback pointer.
5. 5. Monitor drift and learn from feedbackOnline inference records latency, errors, drift, and delayed labels so real time model monitoring and drift detection system retraining is evidence-driven rather than triggered by guesswork.

Deep dives and trade-offs

• Reproducibility and leakage preventionPin real time model monitoring and drift detection system data, feature, code, dependency, and model versions for every run. Use point-in-time joins and quarantine failed quality or privacy checks before training. Keep raw inputs and artifacts immutable so a result can be replayed after a dependency changes.
• Safe promotion and serving contractsSeparate real time model monitoring and drift detection system model registration from deployment and require signed artifacts plus schema compatibility. Use shadow traffic, canaries, rollback pointers, and per-version latency/error budgets. Return model version and feature freshness so clients can explain or reproduce a prediction.
• Drift, feedback, and costMeasure feature drift, prediction drift, label delay, and segment-level quality for real time model monitoring and drift detection system rather than only aggregate accuracy. Sample expensive inference and cap retraining concurrency with an explicit GPU or compute budget. Keep human corrections and delayed labels linked to the original prediction and model version.
• Batch versus online featuresPrefer a shared feature contract with batch backfills and a low-latency online serving path for decisions that need freshness. Two independently defined transformations create training-serving skew and hard-to-debug regressions.
• Synchronous versus asynchronous inferenceKeep interactive real time model monitoring and drift detection system inference synchronous within a strict budget and queue large or expensive jobs. A request path that waits for model loading, enrichment, or retraining turns downstream slowness into an outage.
• Global model versus segment modelsStart with one versioned model and add segment-specific models only when quality or policy evidence justifies the operational cost. Many simultaneously active versions multiply monitoring, rollback, and data-lineage burden.