Notification System — System Design Interview Practice

Interview prompt

Design a multi-channel notification system that delivers email, SMS, and push messages reliably with preferences, templates, retries, provider limits, and observable status.

• Define notification intent, user preferences, templates, delivery attempts, and idempotency as separate domain records.
• Use a durable queue and channel-specific workers to absorb 100M notifications per day without making the API wait for providers.
• Keep provider delivery, retries, dead letters, and analytics off the request acknowledgement path while exposing delivery status.
• Explain opt-out and quiet-hour policy, provider throttling, duplicate suppression, security, and degraded behavior.

Requirements and scale assumptions

• Accept a notification request for one or more users, channels, templates, and data variables.
• Resolve user preferences, consent, quiet hours, locale, and channel eligibility before delivery.
• Render versioned templates and send through email, SMS, and push providers.
• Retry transient provider failures with backoff, enforce provider rate limits, and route exhausted work to a dead-letter queue.
• Expose request and per-channel delivery status, provider message identifiers, and timestamps.
• Support idempotent submission, cancellation before send, scheduled delivery, and authenticated administration.
• Acknowledge an accepted notification request within 100ms under normal load.
• Process 100M notifications per day with burst capacity and independent channel scaling.
• Never send after a durable opt-out or violate channel-specific consent and quiet-hour policy.
• Make provider timeouts and callbacks safe to retry without creating duplicate sends or status regressions.
• Keep a provider outage isolated to its channel and expose pending or degraded status instead of silently dropping work.
• Process 100M notifications per day, roughly 1.2K per second on average, with a planning peak of 10K per second.
• Assume email, SMS, and push have different payload sizes, provider quotas, latency, and retry policies.
• Partition requests and delivery attempts by tenant and recipient, while isolating a broadcast or celebrity recipient from ordinary traffic.
• Retain delivery attempts and consent evidence according to policy; keep rendered payloads and provider responses access-controlled.

• Daily volume: 100M notifications/day — Capacity assumption for queue partitions, worker pools, provider quotas, and storage retention.
• Peak intake: 10K requests/second — Burst planning target that the API and durable queue must absorb without synchronous provider calls.
• API acknowledgement: p99 < 100ms — Budget to validate, persist, and enqueue the request; provider delivery is asynchronous.
• Delivery freshness: p95 < 60s for healthy providers — Example end-to-end target from accepted request to provider acceptance, measured per channel.
• Duplicate rate: < 0.01% retried sends — Operational target for idempotency and provider callback reconciliation, not an end-to-end exactly-once promise.

Key entities

• NotificationRequestrequestId (PK), tenantId, recipientId, templateId, channels, variables, idempotencyKey, status, createdAt

  The durable intent to notify a recipient. It records the caller's request and policy version without storing an unbounded rendered payload.

• UserPreferencerecipientId (PK), channel, category, optIn, quietHours, locale, updatedAt, version

  The current consent and delivery policy used before enqueueing or sending a channel attempt.

• TemplatetemplateId (PK), version, channel, locale, subject, body, variableSchema, status

  An immutable, reviewed channel-specific template version. Rendering uses the version captured by the notification request.

• DeliveryAttemptattemptId (PK), requestId, recipientId, channel, attemptNumber, status, nextRetryAt, providerMessageId, updatedAt

  The idempotent state machine for one channel delivery, including retry timing, provider outcome, and a monotonic status transition.

• ProviderMessageproviderMessageId (PK), providerId, requestId, channel, acceptedAt, status, callbackAt

  External provider evidence used to reconcile timeouts, callbacks, and delivery status without trusting a single response forever.

Data flow

1. 1. Accept and persist notification intentThe API authenticates the tenant, validates template variables and channels, writes NotificationRequest with its idempotency key, and returns an accepted request before contacting a provider.
2. 2. Apply consent and scheduling policyA policy step reads the recipient's preferences, quiet hours, locale, suppression lists, and scheduledAt value, then creates only eligible channel work.
3. 3. Render and enqueue channel workWorkers pin a reviewed Template version, render a bounded payload, create a DeliveryAttempt, and enqueue it to a channel-specific queue with a deduplication key.
4. 4. Send with provider-aware retriesEmail, SMS, and push workers enforce per-provider quotas, call an adapter, classify transient versus permanent errors, and back off or dead-letter without blocking other channels.
5. 5. Reconcile status and measure deliveryProvider callbacks and timeout reconciliation update attempts monotonically; dashboards aggregate delivery latency, failure reason, consent decisions, queue lag, and freshness asynchronously.

Deep dives and trade-offs

• Intent, attempt, and idempotencyNotificationRequest represents what the product asked for; DeliveryAttempt represents one channel state machine. Keeping them separate prevents a provider retry from mutating the original intent. Use tenant plus idempotencyKey and a request hash for intake, then tenant plus requestId plus channel for delivery deduplication. A timeout after provider handoff is unknown, not automatically failed. Reconcile by provider message ID or a provider-specific idempotency key before retrying.
• Preferences, consent, and quiet hoursStore category and channel policy with versions and effective times; a request accepted before an opt-out still needs a send-time policy check. Separate mandatory transactional notices from marketing or engagement notices, and make that classification explicit in the request. A quiet-hour decision can reschedule work, suppress it, or choose another eligible channel, but the outcome should be visible in status and audit data.
• Provider quotas and channel isolationEmail, SMS, and push have different throughput, payload, cost, and callback semantics; use separate queues and worker pools. Token buckets or leaky buckets at provider and tenant scope prevent a retry storm from exhausting a provider quota. Circuit breakers and controlled provider failover should preserve consent and idempotency semantics rather than blindly duplicating a send.
• Retries, dead letters, and unknown outcomesRetry timeouts and rate limits with exponential backoff and jitter; do not retry invalid addresses, revoked tokens, or policy suppressions forever. Persist attempt number, nextRetryAt, error class, and provider evidence so workers can resume after a crash. Dead-letter exhausted or malformed work with an operator replay path that rechecks current consent and template policy.
• Template rendering and versioningValidate variables and output size before enqueueing, pin a template version, and keep rendering bounded and deterministic. Do not let template edits change the meaning of an in-flight request silently; store the version used by each attempt. Sanitize user-provided variables, protect secrets, and restrict administrative template changes with review and audit.
• Delivery status and reconciliationExpose accepted, queued, suppressed, sending, provider-accepted, delivered, failed, and unknown states with per-channel detail. Callbacks are at least once and can arrive out of order; deduplicate event IDs and reject status regressions using conditional transitions. Reconciliation jobs query provider evidence for stale unknown attempts and publish a freshness watermark for dashboards.
• Burst protection and fairnessPartition queues by tenant and recipient class, enforce batch bounds, and reserve worker capacity for transactional messages. A single broadcast should not monopolize all channel workers or provider quota; use weighted fairness and separate bulk capacity. Backpressure should stop or defer intake before queue retention and provider limits are exceeded, with an explicit response contract.
• Privacy and abuse controlsEncrypt recipient addresses, tokens, and rendered content at rest and restrict access to operational roles that need it. Rate-limit senders, detect spam and bounce abuse, and maintain suppression lists without exposing whether a recipient exists. Retain consent evidence and delivery metadata only as long as policy requires, and avoid logging full message bodies by default.
• Synchronous provider calls versus durable enqueueAcknowledge after intent and queue commit; keep provider calls asynchronous so provider latency and outages do not break the product request path. The user sees an accepted or pending state rather than immediate delivery, so status freshness and operational alerts become part of the product.
• One queue versus channel-specific queuesUse a shared intake boundary with channel-specific queues and workers so quotas, retries, and outages remain isolated. More queues require more operations and routing logic; a single queue makes one provider or campaign more likely to create head-of-line blocking.
• Provider failover versus duplicate-send riskFail over only when provider evidence and idempotency semantics are understood; otherwise mark the attempt unknown and reconcile it. Immediate failover after a timeout can send two messages because the first provider may have accepted the request.
• Immediate suppression versus scheduled deliveryRecheck consent and quiet hours immediately before provider handoff and reschedule only when the policy permits it. Checking only at intake can violate a later opt-out; checking only at send can make queue work more expensive but is safer.
• Exact counts versus rebuildable analyticsUse delivery state as the operational record and derive dashboards asynchronously with a freshness watermark. Aggregates can lag or be corrected, so they should not be presented as stronger evidence than the per-attempt state.