Build a Big Data Processing Pipeline — System Design Interview Practice

Interview prompt

Design a batch and streaming data platform that ingests events from many sources, transforms them reliably, and serves fresh analytical results without losing data during backpressure or worker failure.

• Separate immutable raw data, replayable events, stateful stream processing, and query-optimized analytical tables.
• Explain event-time windows, late data, watermarks, deduplication, schema evolution, and exactly-once effects.
• Use one ingestion contract for streaming and batch backfills while isolating noisy tenants and hot keys.
• Define data-quality gates, privacy controls, replay, dead-letter handling, and operational freshness metrics.

Requirements and scale assumptions

• Ingest events from APIs, message brokers, databases, and object-storage batch drops.
• Validate schemas, normalize records, enrich them, and run windowed streaming and historical batch transformations.
• Write raw immutable data and curated analytical tables for dashboards, ad-hoc queries, and exports.
• Expose pipeline configuration, run status, freshness, quality failures, replay, backfill, and cancellation.
• Acknowledge accepted events after durable ingestion, not after every downstream table is updated.
• Target p95 streaming freshness below 10 seconds for healthy partitions and make lag visible.
• Preserve ordering where required by a business key while scaling independent partitions in parallel.
• Make retries, late events, duplicate delivery, schema changes, partial sink failure, and regional recovery safe.
• Ingest 2 million events per second at peak, with records averaging 2 KB and 10x bursts.
• Retain raw data for 90 days in object storage and curated data for years in a columnar warehouse.
• Largest tenants can dominate a partition, so partition keys may require salting and tenant quotas.
• A streaming job has hundreds of partitions and a batch backfill may run thousands of tasks.

• Peak ingest rate: 2M events/s — Drives broker partitions, network capacity, and admission control.
• Streaming freshness: p95 <=10s — Time from source event timestamp to curated-table availability.
• Raw retention: 90 days — Replay and audit window before data moves to a lower-cost tier.
• Quality acceptance: >99.9% valid — Invalid records are quarantined with reason codes rather than silently dropped.

Key entities

• SourcePartitionsourceId, partitionId, cursor, schemaVersion, watermark, status

  Replayable source partition with the last accepted offset and schema contract.

• StreamJobjobId, operatorGraph, partitionKey, checkpointId, parallelism, status

  Versioned streaming computation and its checkpoint lineage.

• CheckpointcheckpointId, jobId, inputOffsets, stateUri, createdAt, status

  Atomic recovery point used to resume stateful operators without duplicating output.

• OutputWindowwindowId, jobId, windowStart, windowEnd, watermark, completeness, resultUri

  Window result that distinguishes final, late, and incomplete data.

Data flow

1. 1. Register sources and contractsConnectors authenticate each source, validate schema compatibility and partition ownership, and persist a source manifest before ingestion.
2. 2. Ingest with backpressureEvents enter partitioned durable buffers; producers receive bounded backpressure while failed records go to a replayable dead-letter path.
3. 3. Process event timeStream operators use checkpoints, watermarks, deduplication, and explicit late-data policy to produce windowed patterns and aggregates.
4. 4. Publish analytical outputsBatch and streaming sinks write versioned results and action intents only after the corresponding checkpoint and quality gates succeed.
5. 5. Recover and reconcileA failed worker resumes from a checkpoint, replays affected partitions, and compares sink counts and watermarks before declaring recovery.

Deep dives and trade-offs

• Exactly-once effects and checkpointsFor the batch and streaming analytics platform, accepted offsets, checkpoint state, and sink writes must advance together. Design for the failure case where a slow partition or failed worker must not silently advance a window or lose an event; keep retries, versions, and repair state explicit. Expose freshness, version, lineage, or audit metadata so operators and clients can distinguish current, pending, and degraded state.
• Watermarks, late data, and backpressureFor the batch and streaming analytics platform, accepted offsets, checkpoint state, and sink writes must advance together. Keep this concern off unrelated request paths and partition it by the batch and streaming analytics platform access key. Expose freshness, version, lineage, or audit metadata so operators and clients can distinguish current, pending, and degraded state.
• Sink correctness and replayFor the batch and streaming analytics platform, accepted offsets, checkpoint state, and sink writes must advance together. Keep this concern off unrelated request paths and partition it by the batch and streaming analytics platform access key. Expose freshness, version, lineage, or audit metadata so operators and clients can distinguish current, pending, and degraded state.
• Event-time correctness versus latencyUse watermarks and bounded lateness for correct windows, then expose provisional results while waiting for late events. Waiting indefinitely for stragglers makes the dashboard unusable; dropping them silently makes aggregates untrustworthy.
• Shared stream engine versus isolated jobsShare the platform control plane but isolate hot or high-value jobs with independent queues and checkpoints. One overloaded topology should not delay unrelated tenants or sinks.
• Reprocessing versus deduplicated sinksMake replay a first-class operation and require sink idempotency keys or transactional commits. Replaying compute without sink deduplication creates duplicate actions and aggregates.