Design an AI-Powered Alert Correlation and Noise Reduction System — System Design Interview Practice

Interview prompt

Design an alert-intelligence platform that normalizes signals, deduplicates and correlates related alerts into incidents, suppresses safely, and learns from operator feedback without hiding important failures.

• Define normalized alert identity, entity and service topology, temporal windows, incident lifecycle, suppression policy, confidence, and evidence.
• Correlate duplicates, flapping, causal chains, and maintenance noise while preserving raw signals and allowing operators to reveal suppressed alerts.
• Separate ingestion and urgent routing from asynchronous enrichment, clustering, learning, and exploratory queries; make correlation replayable.
• Explain false merges and splits, feedback safety, tenant isolation, outage behavior, auditability, observability, and raw-alert fallback.

Requirements and scale assumptions

• Ingest alerts from monitoring tools, normalize identities, deduplicate, correlate related signals into incidents, and route notifications.
• Support topology, temporal, message, and historical features; maintenance windows; expiring suppression; and operator merge, split, acknowledge, resolve, and reveal actions.
• Provide raw-signal access, replay, policy and model versioning, audit history, tenant controls, feedback labels, exports, and recovery after correlation or notification failure.
• Correlate ordinary alerts within 10 seconds while urgent alerts remain visible and routable when enrichment or AI inference is delayed.
• Handle 1 million alerts per hour across thousands of services without a single hot tenant, service, or incident key dominating processing.
• Never lose an accepted raw alert or operator action; make ingestion, correlation, notification, and replay retries idempotent.
• Degrade to raw-alert routing and conservative suppression when topology, feature stores, models, indexes, or notification providers fail.
• Ingest 1 million alerts per hour at peak, with bursts during regional outages and approximately 100,000 incidents per day.
• Track 10,000 services and millions of monitored entities; partition by tenant and time while sharding exceptionally noisy services.
• Retain the immutable raw alert log and operator actions for replay and audit; keep active incident projections and feature windows bounded by retention policy.

• Peak scale: 1M alerts/hour; 10k services — Drives partitioning, admission control, hot-key isolation, and queue capacity during outages.
• Correlation SLO: p95 incident update under 10s — The normal correlation budget; urgent alerts bypass slow enrichment and remain visible immediately.
• Noise-reduction quality: Track false merge, false split, and suppressed-critical rates — Quality must measure safety, not only the percentage of alerts suppressed.
• Durable boundary: Raw alert and operator action committed first — Incidents, suppression decisions, confidence, and notification state are derived and versioned from durable evidence.
• Async boundary: Enrichment, clustering, and model scoring are replayable workers — Slow topology joins, NLP similarity, historical features, and model scoring never block urgent raw-alert acceptance.

Key entities

• RawAlerttenantId, alertId, source, observedAt, fingerprint, severity, entityId, serviceId, labels, payload, receivedAt

  Append-only source record keyed by tenant and alert ID. Store the original payload and a normalized fingerprint so duplicate delivery, replay, and forensic inspection are safe.

• IncidenttenantId, incidentId, state, firstSeen, lastSeen, severity, confidence, rootCauseCandidates, alertCount, modelVersion, policyVersion, updatedAt

  Derived aggregate keyed by tenant and incident ID. Keep evidence references, merge and split history, suppression state, and the latest correlation version rather than copying every raw payload.

• TopologyVersiontenantId, version, serviceEdges, entityOwnership, validFrom, validTo, source, createdAt

  Versioned service and entity relationships let a replay use the topology that was known when the alert occurred and prevent stale ownership from causing unsafe merges.

• CorrelationDecisiontenantId, decisionId, incidentId, alertIds, features, confidence, reasons, policyVersion, modelVersion, createdAt

  Immutable evidence for an attach, merge, split, or suppress decision. Store human-readable reasons and feature references for audit, evaluation, and rollback.

• OperatorActiontenantId, actionId, actorId, incidentId, action, targetAlertIds, expectedVersion, createdAt

  Optimistically versioned command record for acknowledge, resolve, merge, split, reveal, unsuppress, or feedback actions.

• RoutingPolicytenantId, policyId, version, matchRules, suppressionRules, notificationTargets, expiry, updatedBy

  Policy versions define maintenance windows, quiet periods, escalation, and fail-open behavior. A decision records the version it used.

Data flow

1. 1. Accept and normalize the alertThe intake boundary authenticates the source, validates the envelope, normalizes severity and entity identity, derives a deterministic fingerprint, and appends the original payload before acknowledging the sender.
2. 2. Route urgent evidence immediatelyA committed raw alert is eligible for conservative urgent routing without waiting for AI enrichment. Deduplication uses tenant, source, fingerprint, and a bounded time window while retaining every delivery for audit.
3. 3. Correlate asynchronouslyWorkers join the alert with the topology version, temporal window, message features, historical fingerprints, and neighboring incidents; they emit an explainable attach, merge, split, or new-incident decision.
4. 4. Apply suppression and notifyA policy engine evaluates maintenance windows, expiry, severity, confidence, and fail-open rules. It updates the incident projection and sends idempotent notifications, while suppressed alerts remain revealable.
5. 5. Serve, learn, and replayOperators read incident and raw-alert projections, issue versioned actions, and provide feedback. Replay rebuilds correlation decisions with pinned topology, policy, and model versions without mutating the raw log.

Deep dives and trade-offs

• Identity, deduplication, and flappingBuild a stable fingerprint from source identity, alert rule, normalized entity, and selected labels; do not include volatile timestamps or free-form payload fields. Keep every raw delivery while making the derived membership operation idempotent. Use a bounded flap window and explicit recovery events so a recovered service is not silently discarded. Use tenant and time partitions, then isolate a noisy service or outage region with admission control and a dedicated queue.
• Correlation quality and explainabilityCombine temporal overlap, topology adjacency, shared entities, message similarity, and historical co-occurrence; require stronger evidence for cross-service merges than for duplicates. Store candidate edges, feature values or references, reasons, confidence, model version, and policy version so operators can understand and reverse a decision. Measure false merges and false splits separately. Use shadow scoring and replay evaluation before promoting a new model or threshold.
• Safe suppression and urgent routingTreat suppression as a versioned, expiring decision, never deletion. Preserve raw alerts and let operators reveal or unsuppress them from the incident view. Fail open for high severity, low confidence, missing topology, stale policy, or model timeout. Keep urgent routing independent from slow enrichment. Deduplicate notifications by incident transition and policy version, but retain provider attempts and delivery failures for escalation repair.
• Replay, feedback, and model governanceReplay from the raw log with pinned topology, policy, and model versions; write a new decision stream and compare it with the production projection before repair. Treat operator merge, split, reveal, and feedback actions as labeled evidence, protected by optimistic concurrency and audit identity. Monitor drift in alert vocabulary, service topology, incident size, suppression rate, and critical-hidden rate; require rollback controls for model or policy regressions.
• Rules versus learned correlationUse deterministic fingerprints, severity guardrails, and topology constraints as the safety floor; use learned similarity and historical features to rank or suggest relationships. A black-box model that directly suppresses alerts can hide a new failure mode and makes incident review difficult.
• Immediate routing versus complete enrichmentAccept and conservatively route urgent alerts immediately, then refine the incident asynchronously as topology and model features arrive. Waiting for every feature improves grouping in the happy path but increases outage detection latency and creates a dangerous dependency chain.
• Central incident store versus stream-derived viewsKeep raw evidence immutable and apply versioned decisions to an incident aggregate, with searchable projections for responders. A projection-only design makes operator actions, optimistic concurrency, and precise repair semantics harder to guarantee.