Create a Distributed File Transfer System like Bittorrent — System Design Interview Practice

Interview prompt

Design a BitTorrent-like peer-to-peer file distribution system that splits large files into verifiable pieces, discovers healthy peers, and uses available upload bandwidth fairly.

• Make a signed torrent manifest and piece hashes authoritative while peer availability and transfer state remain ephemeral.
• Use a tracker or DHT for discovery, rarest-piece selection for swarm health, and bounded peer sets to avoid connection explosions.
• Verify every piece before assembly, resume partial downloads, and prevent malicious peers from poisoning the swarm.
• Explain NAT traversal, seeding, fairness, abuse controls, privacy, tracker failure, and content takedown.

Requirements and scale assumptions

• Publish a file manifest, discover peers, request pieces, verify hashes, assemble the file, and seed completed pieces.
• Show progress, peer health, piece availability, throughput, retry state, and integrity failures.
• Support private swarms, signed manifests, revocation, trackerless discovery, and abuse or copyright reports.
• Target high aggregate utilization while keeping peer connection and control-message overhead bounded.
• Support millions of swarms through DHT partitioning, tracker sharding, and short-lived peer liveness records.
• Never accept an unverified piece; make manifest publication, piece requests, and resume checkpoints safe to retry.
• Fall back from tracker to DHT and from rarest-piece peers to a limited seed set during partial outages.
• Support 10 million active swarms, 1 million concurrent peers, and files from 100 MB to 10 TB.
• Partition discovery by info-hash and DHT bucket; cap peers per client and isolate very hot releases.
• Persist manifests and moderation decisions while treating peer presence, choking state, and transfer progress as ephemeral.

• Active swarms: 10M peak — Drives DHT buckets, tracker shards, peer limits, and liveness sampling.
• Piece verification: 100% before use — Integrity is stricter than low latency; bad pieces are discarded and retried from another peer.
• Durable boundary: Committed before async — The signed manifest and piece hashes are authoritative; peer availability is a rebuildable view.
• Async boundary: At-least-once workers — Keep peer discovery, liveness, telemetry, and abuse analysis off the file-integrity path.

Key entities

• InteractioninteractionId, actorId, objectId, type, version, occurredAt

  Canonical distributed file transfer system interaction with an idempotency key and ordering version.

• ConnectionSessionsessionId, userId, deviceId, roomKey, lastHeartbeat, status

  Ephemeral but observable distributed file transfer system connection registration used for routing and presence.

• FanoutCursorstreamKey, shard, offset, consumerGroup, updatedAt

  Durable progress marker for distributed file transfer system fan-out and replay.

• DeliveryReceiptinteractionId, recipientId, channel, attempt, status, deliveredAt

  Deduplicated distributed file transfer system delivery state for reconnects, retries, or acknowledgements.

Data flow

1. 1. Accept and commit the interactionThe distributed file transfer system gateway authenticates the actor, validates room or object membership, applies rate limits, and conditionally commits the interaction.
2. 2. Publish an ordered eventAn outbox emits the committed distributed file transfer system transition with an event ID, partition key, sequence, and replay retention.
3. 3. Fan out by partitionConsumers route distributed file transfer system events to connected recipients, durable inboxes, or notification channels without making the origin write wait for every recipient.
4. 4. Resume and reconcile connectionsClients reconnect with a cursor; the distributed file transfer system service replays missed events, deduplicates delivery, and exposes stale or degraded state.
5. 5. Measure latency and recoverOperations tracks distributed file transfer system publish-to-deliver latency, hot partitions, reconnect storms, dropped events, and consumer lag for replay or repair.

Deep dives and trade-offs

• Ordering, idempotency, and hot keysChoose a distributed file transfer system partition key that preserves required order while distributing high-volume rooms, users, or objects. Use event IDs, inboxes, consumer offsets, and conditional state transitions for at-least-once delivery. Split or isolate hot partitions without changing the client-visible sequence contract.
• Reconnect and replay semanticsIssue resumable distributed file transfer system cursors with an expiry and a clear snapshot-plus-delta fallback. Bound replay windows and rebuild from durable state when a cursor is too old. Expose version and freshness so a client can distinguish current, catching up, and degraded state.
• Backpressure and presenceKeep connection heartbeats and ephemeral presence separate from durable distributed file transfer system interactions. Coalesce safe updates, shed low-value work, and protect critical events during reconnect storms. Measure end-to-end delivery, not only broker publish latency.
• Direct fan-out versus pull-based readsUse push for latency-sensitive distributed file transfer system deltas and pull or replay for reconnect, history, and recovery. A push-only design loses state when clients disconnect and a pull-only design wastes latency and bandwidth.
• Per-recipient queues versus shared streamsUse shared partitioned streams with per-recipient cursors where fan-out is large, and isolate exceptional high-fanout objects. A queue per recipient becomes expensive and hard to inspect at large scale.
• Strong ordering versus availabilityGuarantee ordering only within the scope the product needs, such as a room, object, or conversation. Global ordering introduces a bottleneck and still does not solve duplicate delivery or reconnect recovery.