Design an Automated Root Cause Analysis Platform — System Design Interview Practice

Interview prompt

Design an evidence-driven root-cause analysis platform that correlates metrics, logs, traces, topology, deployments, and incidents to rank likely causes while preserving operator trust and raw evidence.

• Define incident windows, signal identity, service topology, deployment/change context, hypotheses, evidence links, confidence, and analyst feedback.
• Correlate temporal and dependency relationships without claiming causality from correlation; handle missing telemetry, topology drift, and noisy signals.
• Separate telemetry ingestion from feature/graph construction, interactive investigation, offline learning, and immutable incident evidence.
• Explain privacy, tenant isolation, feedback bias, replay, failure recovery, observability, and a deterministic evidence-only fallback.

Requirements and scale assumptions

• Ingest and normalize signals, detect incident boundaries, build topology context, rank root-cause hypotheses, and link supporting evidence.
• Provide investigation timelines, deployment correlation, blast-radius views, confidence/limitations, operator annotations, and incident exports.
• Support replay, topology updates, signal deletion, model/version rollback, access controls, and recovery after delayed or missing telemetry.
• Produce an initial evidence-backed hypothesis within five minutes while labeling uncertainty and never hiding the raw incident signals.
• Analyze 100k incidents per day across 50k services without a single hot key or unbounded synchronous work.
• Do not lose committed state; make retries and duplicate events safe.
• Degrade safely when downstream workers, caches, or external dependencies fail.
• 50k services, 100k incidents/day, and 10B telemetry records/day
• Partition by the primary tenant, user, item, or geographic key and isolate hot partitions.
• Keep serving state bounded; retain raw events or durable records for replay and auditing.

• Peak scale: 50k services; 100k incidents/day — Capacity assumption that drives partitioning and backpressure.
• Latency target: initial hypothesis < 5m; raw evidence preserved — User-facing budget for the primary request or read path.
• Durable boundary: Committed before async — Raw telemetry, topology snapshots, and incident records are authoritative; hypotheses are derived evidence.
• Async boundary: At-least-once workers — Keep Use causal inference models (PC algorithm, Granger causality), Implement graph-based propagation analysis, Use anomaly detection across correlated time-series off the synchronous path.

Key entities

• DatasetVersiondatasetId, version, schemaHash, qualityStatus, lineage, createdAt

  Immutable automated root cause analysis platform input version used for reproducible training, evaluation, or replay.

• FeatureSnapshotentityId, featureSetVersion, eventTime, values, sourceWatermarks

  Point-in-time automated root cause analysis platform features with source watermarks so online and offline values can be compared.

• TrainingRunrunId, datasetVersion, codeVersion, metrics, artifactUri, status

  Audited automated root cause analysis platform run that records data, code, dependency, and evaluation lineage.

• ModelVersionmodelId, version, stage, schema, qualityGates, endpoint

  A promotable automated root cause analysis platform model version with rollout state, contract, and rollback metadata.

Data flow

1. 1. Register and validate training dataThe automated root cause analysis platform gateway records an immutable dataset version, schema, lineage, quality status, and privacy disposition.
2. 2. Build point-in-time featuresFeature workers join automated root cause analysis platform inputs using event-time watermarks, prevent leakage, and publish the same feature contract for training and serving.
3. 3. Train and evaluate asynchronouslyThe orchestrator schedules automated root cause analysis platform runs with checkpointed artifacts, reproducible environments, and metrics tied to the exact input versions.
4. 4. Gate and serve a model versionA registry compares automated root cause analysis platform quality, bias, safety, and compatibility gates before canary or production rollout with an immediate rollback pointer.
5. 5. Monitor drift and learn from feedbackOnline inference records latency, errors, drift, and delayed labels so automated root cause analysis platform retraining is evidence-driven rather than triggered by guesswork.

Deep dives and trade-offs

• Reproducibility and leakage preventionPin automated root cause analysis platform data, feature, code, dependency, and model versions for every run. Use point-in-time joins and quarantine failed quality or privacy checks before training. Keep raw inputs and artifacts immutable so a result can be replayed after a dependency changes.
• Safe promotion and serving contractsSeparate automated root cause analysis platform model registration from deployment and require signed artifacts plus schema compatibility. Use shadow traffic, canaries, rollback pointers, and per-version latency/error budgets. Return model version and feature freshness so clients can explain or reproduce a prediction.
• Drift, feedback, and costMeasure feature drift, prediction drift, label delay, and segment-level quality for automated root cause analysis platform rather than only aggregate accuracy. Sample expensive inference and cap retraining concurrency with an explicit GPU or compute budget. Keep human corrections and delayed labels linked to the original prediction and model version.
• Batch versus online featuresPrefer a shared feature contract with batch backfills and a low-latency online serving path for decisions that need freshness. Two independently defined transformations create training-serving skew and hard-to-debug regressions.
• Synchronous versus asynchronous inferenceKeep interactive automated root cause analysis platform inference synchronous within a strict budget and queue large or expensive jobs. A request path that waits for model loading, enrichment, or retraining turns downstream slowness into an outage.
• Global model versus segment modelsStart with one versioned model and add segment-specific models only when quality or policy evidence justifies the operational cost. Many simultaneously active versions multiply monitoring, rollback, and data-lineage burden.