Distributed Job Scheduler — System Design Interview Practice

Interview prompt

Design a multi-tenant job scheduling service that runs one-time and recurring jobs reliably, even when schedules spike, workers fail, or a dispatcher is restarted.

• Separate schedule management, due-work detection, dispatch, and execution.
• Explain timing semantics, duplicate delivery, retries, time zones, and missed runs.
• Use durable state and leases so dispatcher and worker failures recover safely.
• Make tenant fairness, backpressure, idempotency, and operational visibility explicit.

Requirements and scale assumptions

• Create, update, pause, resume, and delete one-time or recurring schedules.
• Run HTTP, container, or queue-backed jobs with configurable timeout and retry policy.
• Support cron-like expressions, time zones, start/end windows, and misfire policy.
• Expose execution history, current state, manual run, cancellation, and dead-letter replay.
• Isolate tenants with quotas, concurrency limits, authorization, and audit events.
• Provide at-least-once execution with a clear idempotency contract; exactly-once side effects are the job owner's responsibility.
• Target p99 dispatch lateness below 10 seconds for jobs within the normal capacity envelope.
• Survive dispatcher, worker, queue, and storage node failures without silently losing a due job.
• Keep schedule writes strongly consistent while allowing dashboards and metrics to be eventually consistent.
• Apply backpressure and tenant fairness instead of allowing one customer to exhaust all workers.
• 10 million active schedules, with 100 million executions per day.
• Average execution rate is about 1.2K jobs/second; peak bursts reach 10K jobs/second.
• Most schedules are recurring and produce a small schedule row plus one execution record per run.
• Jobs are external work: median runtime is 5 seconds, with a 15-minute maximum lease and timeout.

• Average execution rate: ~1.2K/s — 100M daily executions divided across a 24-hour day.
• Peak dispatch rate: ~10K/s — Burst budget for aligned cron boundaries and catch-up work.
• Active schedules: 10M — The due-time index must be sharded; a full-table scan is not viable.
• Execution history: ~36.5B/year — Use retention tiers and archive old records instead of keeping all history hot.

Key entities

• SchedulescheduleId, tenantId, cron, timezone, nextRunAt, version, status

  Versioned schedule definition and next-run state.

• DispatchLeasescheduleId, partition, leaseOwner, leaseUntil, claimedAt, status

  Fenced ownership of due-schedule dispatch.

• ExecutionAttemptexecutionId, scheduleId, attempt, runKey, workerId, startedAt, status

  Idempotent job execution attempt.

• DeadLetterexecutionId, reason, attempts, payloadRef, createdAt, resolvedAt

  Terminal or operator-reviewed failed execution.

Data flow

1. 1. Create or update a scheduleThe service validates timezone, cron, retry, concurrency, and tenant policy, then atomically updates schedule state and its due-time index.
2. 2. Discover due schedulesPartition owners scan bounded due-time buckets, acquire leases, and fence stale dispatchers before producing execution messages.
3. 3. Run with lease and idempotencyWorkers claim an execution, renew its lease, use a stable run key, and record heartbeats and terminal status.
4. 4. Retry or dead-letterTransient errors are rescheduled with bounded backoff; exhausted or non-retryable failures enter a tenant-visible dead-letter queue.
5. 5. Recover dispatcher gapsA reconciler compares schedules, leases, queue messages, and executions to repair missed dispatch without creating a duplicate run.

Deep dives and trade-offs

• Due-time indexing and dispatcher leasesFor the multi-tenant job scheduler, each scheduled occurrence creates at most one logical execution while allowing safe attempts. Design for the failure case where dispatcher restart, clock skew, worker loss, and retry storms must not duplicate or starve jobs; keep retries, versions, and repair state explicit. Expose freshness, version, lineage, or audit metadata so operators and clients can distinguish current, pending, and degraded state.
• Exactly-once run intentFor the multi-tenant job scheduler, each scheduled occurrence creates at most one logical execution while allowing safe attempts. Keep this concern off unrelated request paths and partition it by the multi-tenant job scheduler access key. Expose freshness, version, lineage, or audit metadata so operators and clients can distinguish current, pending, and degraded state.
• Retries, fairness, and recoveryFor the multi-tenant job scheduler, each scheduled occurrence creates at most one logical execution while allowing safe attempts. Keep this concern off unrelated request paths and partition it by the multi-tenant job scheduler access key. Expose freshness, version, lineage, or audit metadata so operators and clients can distinguish current, pending, and degraded state.
• Polling due indexes versus timer queuesUse a durable due-time index with partition leases and bounded polling; add timer services only when scale requires it. A single timer coordinator becomes a failover bottleneck; unbounded polling burns capacity.
• At-most-once versus at-least-once executionUse at-least-once dispatch with stable run keys and application-level idempotency. At-most-once can silently lose a job during a lease or worker failure.
• Strict fairness versus throughputReserve tenant quotas and weighted queues while allowing controlled borrowing of idle capacity. Pure global FIFO lets one noisy tenant dominate; rigid isolation wastes capacity.