Design a Smart SLA Monitoring and Prediction System — System Design Interview Practice

Interview prompt

Design an SLA intelligence platform that computes service-level indicators and error budgets, predicts breaches across dependency chains, and coordinates proactive mitigation.

• Define SLI event contracts, SLO windows, availability/latency aggregation, exclusions, ownership, and composite dependency semantics.
• Compute multi-window burn rates and remaining error budget, forecast breach risk, and avoid false certainty when samples are sparse or delayed.
• Separate telemetry ingestion from query/alert paths; make rollups, SLO definitions, and forecast versions reproducible and replayable.
• Explain alert deduplication, maintenance windows, remediation approvals, missing data, retention, observability, and degraded dashboards.

Requirements and scale assumptions

• Ingest request, latency, error, and dependency events; calculate SLI windows, SLO compliance, burn rates, and error budgets.
• Expose service and composite-SLA dashboards with forecast confidence, freshness, drill-down evidence, and historical comparisons.
• Create routed alerts and mitigation recommendations, support SLO changes/versioning, replay, and reconciliation after telemetry gaps.
• Update SLO aggregates within one minute and flag high-confidence breach risk at least 15 minutes ahead when evidence supports it.
• Scale to millions of telemetry events per minute and large dependency graphs without a single hot key or unbounded synchronous work.
• Do not lose committed state; make retries and duplicate events safe.
• Degrade safely when downstream workers, caches, or external dependencies fail.
• Millions of events/minute across 10k services and 100k dependency edges
• Partition by the primary tenant, user, item, or geographic key and isolate hot partitions.
• Keep serving state bounded; retain raw events or durable records for replay and auditing.

• Peak scale: 10k services; 100k dependency edges — Capacity assumption that drives partitioning and backpressure.
• Latency target: aggregates < 1m; forecast lead >= 15m — User-facing budget for the primary request or read path.
• Durable boundary: Committed before async — Raw SLI events and versioned SLO definitions are authoritative; rollups and forecasts are derived.
• Async boundary: At-least-once workers — Keep Use burn-rate based alerting for SLO monitoring, Implement time-series forecasting for breach prediction, Use exponential smoothing for error budget projections off the synchronous path.

Key entities

• SliWindowserviceId, window, requestCount, errorCount, latencyQuantiles, watermark

  Event-time SLI evidence for smart sla monitoring and prediction system.

• SloVersionsloId, serviceIds, objective, windowPolicy, budget, version

  Versioned objective and error-budget policy for a service or dependency chain.

• BreachForecastforecastId, sloId, horizon, probability, confidence, modelVersion

  Reproducible prediction of a future SLO breach with evidence references.

• MitigationRecommendationrecommendationId, forecastId, action, evidenceRefs, approval, status

  Policy-gated recommendation that does not silently mutate production.

Data flow

1. 1. Ingest request and dependency evidenceThe smart sla monitoring and prediction system gateway validates service ownership, event time, outcome, latency, trace links, and telemetry schema.
2. 2. Compute SLIs and error budgetsWindow workers calculate availability, latency, burn rate, and composite dependency health with watermarks.
3. 3. Predict future SLO breachesThe predictor joins versioned SLI windows, topology, seasonality, and current burn to produce confidence-scored forecasts.
4. 4. Recommend and route mitigationPolicy checks turn a forecast into an evidence-backed recommendation, approval request, or deduplicated alert.
5. 5. Explain, replay, and improveOperators inspect smart sla monitoring and prediction system evidence, correct telemetry gaps, replay windows, and evaluate forecast calibration over time.

Deep dives and trade-offs

• SLI and error-budget correctnessDefine request population, good-event rules, latency buckets, aggregation windows, and exclusion policy explicitly. Keep event-time watermarks and late-data correction separate from dashboard freshness. Version SLO changes so historical budget burn is not recomputed under a new objective.
• Dependency-chain predictionModel ownership and dependency edges as versioned evidence, not an unbounded graph lookup at alert time. Calibrate forecast confidence by service, horizon, incident class, and missing telemetry. Distinguish a predicted breach from a confirmed SLO violation and show the evidence behind both.
• Safe proactive mitigationRequire policy, blast-radius, cooldown, and approval gates before any operational action. Keep notification delivery and recommendation state independent from the SLI calculation commit. Measure false positives, missed breaches, acknowledgement latency, and mitigation outcome.
• Exact windows versus approximate streamingUse exact calculations for contractual SLOs and bounded approximations only for early warning or exploratory views. An approximate burn rate must expose its error and cannot silently drive a contractual violation decision.
• Prediction versus threshold alertsKeep threshold alerts authoritative and use forecasts to prioritize investigation or approved prevention. A forecast should not page or mutate production without confidence, evidence, and policy gates.
• Central topology versus local ownershipCentralize dependency discovery and freshness while letting service owners define SLI and SLO contracts. A stale dependency graph can produce confident but incorrect chain forecasts.