Build a Real-time Data Streaming Pipeline — System Design Interview Practice

Interview prompt

Design a high-throughput streaming pipeline with durable ingestion, partitioned processing, sub-second projections, replay, backpressure, and safe sink delivery for millions of events per second.

• Define event envelope, ordering key, partitioning, retention, consumer offsets, watermark, replay, and sink delivery semantics.
• Use batching/compression and bounded state; handle hot partitions, duplicates, out-of-order events, poison messages, and consumer backpressure.
• Separate durable append from transformations, materialized views, analytical queries, and external side effects with idempotent sinks.
• Explain checkpoint recovery, schema evolution, data loss policy, quotas, observability, and stale/degraded projections.

Requirements and scale assumptions

• Accept, validate, partition, retain, transform, and publish events to consumers and analytical projections with lag metadata.
• Support replay from offsets, multiple consumer groups, schema versions, filtering, enrichment, dead-letter queues, and sink acknowledgments.
• Make event and side-effect processing retry-safe, support retention/deletion, checkpoint restore, and recovery after broker or worker failure.
• Publish common projections with p95 under one second while preserving durable events for replay and audit.
• Ingest 5M events per second at peak without a single hot key or unbounded synchronous work.
• Do not lose committed state; make retries and duplicate events safe.
• Degrade safely when downstream workers, caches, or external dependencies fail.
• 5M events/second peak, 10k consumers, and multi-day retention
• Partition by the primary tenant, user, item, or geographic key and isolate hot partitions.
• Keep serving state bounded; retain raw events or durable records for replay and auditing.

• Peak scale: 5M events/s; 10k consumers — Capacity assumption that drives partitioning and backpressure.
• Latency target: projection p95 < 1s; replayable log — User-facing budget for the primary request or read path.
• Durable boundary: Committed before async — The durable event log and consumer checkpoints are authoritative; transformations and projections are derived.
• Async boundary: At-least-once workers — Keep Kinesis Data Streams for ingestion, Kinesis Data Analytics for SQL queries, Lambda for data transformation off the synchronous path.

Key entities

• SourcePartitionsourceId, partitionId, cursor, schemaVersion, watermark, status

  Replayable real time data streaming pipeline source evidence and ingestion cursor.

• SchemaVersiondatasetId, version, compatibility, owner, effectiveAt, status

  Governed real time data streaming pipeline contract used to validate producers and consumers.

• ProcessingRunrunId, inputWatermark, checkpoint, qualityStatus, codeVersion, status

  Checkpointed real time data streaming pipeline processing attempt with quality and lineage metadata.

• AnalyticalDatasetdatasetId, partition, watermark, schemaVersion, qualityStatus, location

  Curated real time data streaming pipeline serving partition with freshness and quality state.

Data flow

1. 1. Register sources and contractsThe real time data streaming pipeline catalog records owners, schemas, compatibility rules, retention, lineage, and partitioning before data is accepted.
2. 2. Ingest with backpressureConnectors checkpoint real time data streaming pipeline source cursors, validate schema and deduplication keys, and slow producers when downstream capacity is exhausted.
3. 3. Process event time with checkpointsStream or batch engines compute real time data streaming pipeline transformations using watermarks, late-data policy, state checkpoints, and deterministic code versions.
4. 4. Publish quality-gated datasetsOnly real time data streaming pipeline outputs that pass completeness, freshness, validity, and privacy checks become visible to analytical consumers.
5. 5. Serve, replay, and reconcileConsumers read bounded partitions with freshness metadata while operators replay failed real time data streaming pipeline ranges and compare output checksums.

Deep dives and trade-offs

• Schema evolution and data qualityVersion real time data streaming pipeline contracts and make compatibility rules explicit for every producer and consumer. Quarantine malformed partitions instead of poisoning the whole dataset. Track row counts, null rates, duplicates, distribution changes, and policy violations by partition.
• Watermarks, late data, and exactly-once effectsUse source cursors and event-time watermarks for real time data streaming pipeline progress, not wall-clock assumptions. Make checkpoints, output keys, and sink commits retry-safe under at-least-once delivery. Document how late events revise windows, aggregates, or snapshots.
• Replay, lineage, and costKeep immutable real time data streaming pipeline raw evidence and code or schema versions so failed outputs can be reproduced. Separate hot serving storage from cold retention and cap replay concurrency. Measure freshness, backlog, compute cost, storage growth, and quality-gate failure rate.
• Streaming versus batchUse streaming for freshness-critical real time data streaming pipeline paths and batch for backfills, compaction, and expensive recomputation. Forcing every workload into streaming makes state, replay, and cost harder to operate.
• Raw retention versus curated-only storageRetain enough immutable raw evidence for replay, audit, and correction, then tier or expire it according to policy. Without raw evidence, a bad transformation can require an unreproducible emergency fix.
• Central warehouse versus domain-owned datasetsCentralize governance and discovery while letting domain owners own contracts and quality signals. A single team owning every transformation becomes a delivery bottleneck and hides data ownership.