Design an On-Call Escalation System — System Design Interview Practice

Interview prompt

Design a reliable on-call scheduling and escalation service that resolves current ownership across rotations, sends multi-channel alerts, escalates on timeout, and preserves an auditable incident timeline.

• Define schedules, rotations, overrides, time zones, holidays, teams, escalation policies, incident acknowledgments, and notification attempts.
• Resolve one deterministic responder for a point in time, handle handoffs and concurrent edits, and never silently drop an urgent alert.
• Separate schedule reads from alert intake, delivery workers, provider callbacks, audit history, and escalation timers; make retries idempotent.
• Explain provider outage, quiet hours, deduplication, privacy, access controls, observability, and emergency fallback contacts.

Requirements and scale assumptions

• Create teams/schedules, define rotations and overrides, resolve current on-call ownership, ingest incidents, and send/escalate notifications.
• Support acknowledgment, snooze, handoff, maintenance windows, multi-channel delivery, delivery receipts, and escalation timelines.
• Make schedule changes versioned, retries deduplicated, contacts auditable, personal data controlled, and missed notifications recoverable.
• Resolve ownership immediately and deliver or escalate critical alerts within the configured policy window with visible delivery state.
• Handle 100k incidents per day and large notification bursts without a single hot key or unbounded synchronous work.
• Do not lose committed state; make retries and duplicate events safe.
• Degrade safely when downstream workers, caches, or external dependencies fail.
• 100k incidents/day, 50k responders, and 10 notification providers
• Partition by the primary tenant, user, item, or geographic key and isolate hot partitions.
• Keep serving state bounded; retain raw events or durable records for replay and auditing.

• Peak scale: 100k incidents/day; 50k responders — Capacity assumption that drives partitioning and backpressure.
• Latency target: critical delivery/escalation < policy window — User-facing budget for the primary request or read path.
• Durable boundary: Committed before async — Versioned schedules, incident records, and delivery attempts are authoritative; timers and status views are derived.
• Async boundary: At-least-once workers — Keep Scheduling engine for rotations, Retry logic with escalation, Multiple notification channels off the synchronous path.

Key entities

• OperationalSignalsignalId, source, serviceId, severity, fingerprint, occurredAt

  Deduplicated signal used by on call escalation system grouping and policy.

• IncidentincidentId, serviceIds, severity, status, owner, version

  Versioned incident or operational case.

• ResponsePolicypolicyId, scope, schedule, suppression, escalation, version

  Auditable routing, suppression, or capacity policy.

• DecisionRecorddecisionId, incidentId, actor, evidenceRefs, action, createdAt

  Timeline entry for an escalation, capacity decision, or remediation.

Data flow

1. 1. Ingest and normalize signalsThe on call escalation system gateway validates source, service, ownership, fingerprint, severity, and deduplication before publishing a signal.
2. 2. Group with topology and policyCorrelation uses dependency, owner, schedule, and suppression policy to create an explainable incident or capacity case.
3. 3. Commit operational stateThe incident service conditionally commits severity, ownership, acknowledgements, and timeline evidence.
4. 4. Escalate and notify asynchronouslyWorkers schedule responders, reminders, and capacity actions with leases, quiet hours, and deduplicated delivery.
5. 5. Learn and repairAnalytics measures noise, response, SLO, and capacity outcomes while replay workers repair missed events or stale policy.

Deep dives and trade-offs

• Grouping without hiding riskUse topology, fingerprints, time windows, and causal hints to group related on call escalation system signals. Keep raw signals and evidence links so operators can split an incorrect group. Never let suppression remove critical severity or prevent an audit trail.
• Escalation and schedulesVersion policies, schedules, quiet hours, and fallback responders; make transitions idempotent. Track acknowledgement and notification receipts separately from incident state. Bound retries and use a dead-letter or manual escalation path for provider failures.
• Noise and capacity qualityMeasure alert-to-incident ratio, duplicate rate, missed critical signals, acknowledgement latency, and action success. Use feedback and post-incident review to tune policies without silently changing historical decisions. Keep capacity forecasts and recommendations distinct from autonomous production changes.
• Suppression versus visibilitySuppress duplicate delivery but preserve raw signals, evidence, and a reason for grouping. Deleting noisy signals makes it impossible to prove that a critical event was missed.
• Synchronous page versus queued deliveryCommit incident state first and queue pages or reminders with delivery receipts. Blocking incident creation on a notification provider turns provider outages into incident loss.
• Automatic remediation versus human controlAutomate bounded, reversible actions and require approval for high-blast-radius changes. An incorrect correlation can make automatic remediation amplify an outage.