Design a Distributed Stream Processing System like Kafka — System Design Interview Practice

Interview prompt

Design a Kafka-like distributed log that stores ordered records durably, lets independent consumer groups replay streams, and scales partitions and replicas across brokers.

• Define topics, partitions, offsets, retention, replication, leader election, producer acknowledgements, and consumer-group ownership.
• Preserve order within a partition while scaling throughput through keys, batching, compression, and parallel consumers.
• Make rebalances, duplicate delivery, offset commits, broker loss, and log replication observable and recoverable.
• Explain hot partitions, schema evolution, replay, compaction, cross-region replication, and tenant quotas.

Requirements and scale assumptions

• Create topics, publish keyed records, consume through groups, commit offsets, replay ranges, and compact selected keys.
• Expose partition watermarks, consumer lag, replica health, under-replicated partitions, throughput, and retention state.
• Support ACLs, schema validation, quotas, topic configuration, consumer resets, deletion policy, and cross-region mirroring.
• Target p95 produce-to-consume latency below 100 ms for healthy partitions and expose lag when consumers fall behind.
• Handle 10 million records per second with partitioned logs, batching, compression, and replicated broker disks.
• Do not lose records acknowledged at the configured durability level; make offset commits and consumers idempotent.
• Keep healthy partitions available during broker failure and throttle producers rather than exhausting storage.
• Operate 10,000 topics, 100,000 partitions, and 10 million records per second across a multi-broker cluster.
• Partition by topic and stable record key; detect hot keys and isolate high-volume topics.
• Retain recent logs for replay, compact eligible topics, and archive older segments without blocking leaders.

• Peak scale: Support high throughput (millions of messages/sec) — Capacity assumption that drives partitioning and backpressure.
• Latency target: Low latency for message delivery — User-facing budget for the primary request or read path.
• Durable boundary: Committed before async — The source of truth is Publish and subscribe to streams of records; Store streams durably and fault-tolerantly.
• Async boundary: At-least-once workers — Keep Partition data for parallelism, Replication for fault tolerance, Commit log for durability off the synchronous path.

Key entities

• InteractioninteractionId, actorId, objectId, type, version, occurredAt

  Canonical distributed stream processing system interaction with an idempotency key and ordering version.

• ConnectionSessionsessionId, userId, deviceId, roomKey, lastHeartbeat, status

  Ephemeral but observable distributed stream processing system connection registration used for routing and presence.

• FanoutCursorstreamKey, shard, offset, consumerGroup, updatedAt

  Durable progress marker for distributed stream processing system fan-out and replay.

• DeliveryReceiptinteractionId, recipientId, channel, attempt, status, deliveredAt

  Deduplicated distributed stream processing system delivery state for reconnects, retries, or acknowledgements.

Data flow

1. 1. Accept and commit the interactionThe distributed stream processing system gateway authenticates the actor, validates room or object membership, applies rate limits, and conditionally commits the interaction.
2. 2. Publish an ordered eventAn outbox emits the committed distributed stream processing system transition with an event ID, partition key, sequence, and replay retention.
3. 3. Fan out by partitionConsumers route distributed stream processing system events to connected recipients, durable inboxes, or notification channels without making the origin write wait for every recipient.
4. 4. Resume and reconcile connectionsClients reconnect with a cursor; the distributed stream processing system service replays missed events, deduplicates delivery, and exposes stale or degraded state.
5. 5. Measure latency and recoverOperations tracks distributed stream processing system publish-to-deliver latency, hot partitions, reconnect storms, dropped events, and consumer lag for replay or repair.

Deep dives and trade-offs

• Ordering, idempotency, and hot keysChoose a distributed stream processing system partition key that preserves required order while distributing high-volume rooms, users, or objects. Use event IDs, inboxes, consumer offsets, and conditional state transitions for at-least-once delivery. Split or isolate hot partitions without changing the client-visible sequence contract.
• Reconnect and replay semanticsIssue resumable distributed stream processing system cursors with an expiry and a clear snapshot-plus-delta fallback. Bound replay windows and rebuild from durable state when a cursor is too old. Expose version and freshness so a client can distinguish current, catching up, and degraded state.
• Backpressure and presenceKeep connection heartbeats and ephemeral presence separate from durable distributed stream processing system interactions. Coalesce safe updates, shed low-value work, and protect critical events during reconnect storms. Measure end-to-end delivery, not only broker publish latency.
• Direct fan-out versus pull-based readsUse push for latency-sensitive distributed stream processing system deltas and pull or replay for reconnect, history, and recovery. A push-only design loses state when clients disconnect and a pull-only design wastes latency and bandwidth.
• Per-recipient queues versus shared streamsUse shared partitioned streams with per-recipient cursors where fan-out is large, and isolate exceptional high-fanout objects. A queue per recipient becomes expensive and hard to inspect at large scale.
• Strong ordering versus availabilityGuarantee ordering only within the scope the product needs, such as a room, object, or conversation. Global ordering introduces a bottleneck and still does not solve duplicate delivery or reconnect recovery.