Design a Cost-Optimized Architecture for Batch Processing — System Design Interview Practice

Interview prompt

Design a cost-optimized batch-processing platform that accepts large workloads, schedules them across elastic workers, produces durable results, and keeps compute spend low without sacrificing correctness or predictable completion times.

• Separate durable job intent and input data from ephemeral compute.
• Use queues, capacity-aware scheduling, checkpointing, retries, and idempotent outputs.
• Optimize cost with spot or preemptible capacity, autoscaling, rightsizing, and storage lifecycle policies.
• Define completion SLAs, fairness, tenant quotas, observability, and cheap-capacity fallback behavior.

Requirements and scale assumptions

• Submit, validate, pause, cancel, retry, and inspect tenant-owned batch jobs.
• Support dependencies, priorities, resource requirements, deadlines, retries, checkpoints, and partial progress.
• Stage inputs, execute tasks on elastic workers, persist outputs, and expose logs, metrics, and completion status.
• Allow budgets, quotas, regions, data-retention policies, and maximum completion windows.
• A submitted job and terminal state must be durable; duplicate execution must not duplicate committed outputs.
• Meet a configurable completion SLA, such as 95% of standard jobs completing within 30 minutes.
• Minimize cost while maintaining isolation, fairness, data security, and urgent-work capacity.
• Tolerate worker preemption, zone failure, queue backlog, malformed tasks, slow tenants, and object-store throttling.
• 10,000 tenants submit 1 million jobs per day, with 20,000 runnable tasks at peak.
• Inputs and outputs total 5 PB per month; most data is immutable and lifecycle-tiered.
• Use on-demand capacity for deadlines and spot or preemptible capacity for retryable workloads.
• Jobs range from seconds to hours; avoid one queue or database partition becoming the bottleneck.

• Peak runnable tasks: ~20K — Drives queue partitioning, scheduler throughput, and worker-pool autoscaling.
• Completion SLA: 95% <=30 min — Cheap capacity is used until deadline risk requires on-demand capacity.
• Cost target: Spot-first — Prefer preemptible capacity for checkpointable work and fall back when SLA risk rises.
• Data growth: 5PB/month — Requires lifecycle tiers, locality-aware scheduling, and bounded intermediate data.

Key entities

• BatchJobjobId, tenantId, inputUris, taskGraph, priority, deadline, status

  Durable job specification and lifecycle state.

• TaskAttemptattemptId, jobId, taskId, leaseUntil, workerClass, checkpointUri, status

  Retry-safe attempt with lease, resource class, and checkpoint.

• ArtifactManifestjobId, artifactId, uri, contentHash, size, retention, status

  Content-addressed output manifest used by downstream consumers.

• CapacityDecisiondecisionId, jobId, workerClass, spotShare, estimatedCost, risk, policyVersion

  Auditable cost, capacity, and interruption decision.

Data flow

1. 1. Accept a batch jobThe gateway validates input manifests, dependency graph, deadline, budget, tenant quota, and idempotency before recording the job.
2. 2. Plan cost-aware capacityThe planner chooses worker classes, spot or on-demand mix, retry budget, and checkpoint cadence while protecting the completion SLO.
3. 3. Lease and execute tasksPartition owners lease tasks to elastic workers; attempts checkpoint durable progress and return capacity on completion or interruption.
4. 4. Publish content-addressed artifactsReducers and validators write immutable outputs, verify hashes, and publish a manifest only after all required tasks pass.
5. 5. Recover and control spendThe reconciler retries safe attempts, restores from checkpoints, drains bad capacity, and records actual cost against the budget.

Deep dives and trade-offs

• Checkpointing and interruption recoveryFor the cost-optimized batch-processing platform, a job is complete only when required artifacts and billing evidence are durable. Design for the failure case where spot interruption, worker loss, or a retry storm must not duplicate outputs or exceed the budget; keep retries, versions, and repair state explicit. Expose freshness, version, lineage, or audit metadata so operators and clients can distinguish current, pending, and degraded state.
• Cost-aware schedulingFor the cost-optimized batch-processing platform, a job is complete only when required artifacts and billing evidence are durable. Keep this concern off unrelated request paths and partition it by the cost-optimized batch-processing platform access key. Expose freshness, version, lineage, or audit metadata so operators and clients can distinguish current, pending, and degraded state.
• Artifact correctness and retentionFor the cost-optimized batch-processing platform, a job is complete only when required artifacts and billing evidence are durable. Keep this concern off unrelated request paths and partition it by the cost-optimized batch-processing platform access key. Expose freshness, version, lineage, or audit metadata so operators and clients can distinguish current, pending, and degraded state.
• Spot savings versus completion predictabilityUse spot capacity for checkpointable tasks and reserve on-demand capacity for deadlines or non-interruptible stages. Maximizing spot usage can turn interruptions into missed deadlines and higher retry cost.
• Fine-grained tasks versus scheduling overheadChoose task granularity from checkpoint cost, skew, and worker startup time rather than maximizing parallelism. Tiny tasks waste scheduler and object-store overhead; huge tasks make recovery expensive.
• Immediate artifact cleanup versus replayabilityRetain manifests and checkpoints through the job retention window, then tier or delete content by policy. Deleting intermediate state too early forces a full rerun and can cost more than storage.