Design a File Downloader Library from Frontend to Backend — System Design Interview Practice

Interview prompt

Design a frontend-to-backend file downloader that supports resumable range requests, progress, integrity checks, cancellation, retries, and safe handling of changing or unauthorized resources.

• Define a download manifest with URL, length, ETag or version, chunk ranges, checksums, and destination state.
• Use bounded concurrent range requests, resumable checkpoints, request cancellation, and exponential backoff.
• Validate content length, ETag, hash, and safe filename before assembling the file; restart when the source changes.
• Explain signed URLs, redirects, quotas, partial responses, disk exhaustion, browser limits, and corrupted chunks.

Requirements and scale assumptions

• Create a download, discover range support, fetch chunks, verify them, assemble the destination, and report progress.
• Support pause, resume, cancellation, retry, concurrency adjustment, checksum verification, and changed-source detection.
• Protect credentials and signed URLs, prevent path traversal, enforce file-size limits, and clean up partial state.
• Resume after network interruption without restarting completed ranges and expose progress within one chunk interval.
• Bound concurrent connections and disk buffers per download, user, origin, and browser tab.
• Persist checkpoints and verify every chunk; make retries safe and never overwrite a valid destination silently.
• Fall back to one stream or pause with a resumable state when range support, storage, or the origin is degraded.
• Support 100,000 concurrent downloads, files from 1 MB to 1 TB, and unreliable mobile networks.
• Partition state by download ID and origin; cap chunks, sockets, memory, and disk reservations per user.
• Retain only resumable manifests and checksums; clean partial chunks after an explicit idle retention window.

• Peak scale: Concurrent chunk downloads — Capacity assumption that drives partitioning and backpressure.
• Latency target: Handle network interruptions — User-facing budget for the primary request or read path.
• Durable boundary: Committed before async — The source of truth is Download files from URLs; Resume interrupted downloads.
• Async boundary: At-least-once workers — Keep HTTP Range requests for chunked downloads, Checksum validation (MD5, SHA), Thread pool for concurrent downloads off the synchronous path.

Key entities

• MediaAssetassetId, ownerId, sourceUri, checksum, privacy, status

  Canonical uploaded file downloader library from frontend to backend asset and lifecycle state.

• MediaRenditionassetId, profile, codec, uri, checksum, status

  Derived file downloader library from frontend to backend output identified by a deterministic profile and content hash.

• PlaybackSessionsessionId, assetId, viewerId, entitlementVersion, edgeRegion, expiresAt

  Short-lived file downloader library from frontend to backend access session that binds authorization to delivery.

• ProcessingJobjobId, assetId, operation, attempt, checkpoint, status

  Retry-safe file downloader library from frontend to backend processing job with checkpoints and per-rendition progress.

Data flow

1. 1. Reserve a resumable uploadThe file downloader library from frontend to backend gateway authenticates the owner, reserves metadata, validates size and checksum, and returns a scoped upload URL.
2. 2. Commit and verify the sourceA completion callback verifies the file downloader library from frontend to backend object, records an immutable checksum, and publishes a processing job only once.
3. 3. Process renditions asynchronouslyWorkers execute file downloader library from frontend to backend transforms with deterministic profiles, checkpointing, bounded retries, and a dead-letter path for corrupt inputs.
4. 4. Publish an entitlement-aware manifestA manifest projection exposes only completed file downloader library from frontend to backend renditions and carries policy, checksum, and freshness metadata.
5. 5. Deliver, invalidate, and recoverCDN delivery is protected by expiring URLs and revocation signals; failed file downloader library from frontend to backend jobs and stale manifests are replayable without duplicating outputs.

Deep dives and trade-offs

• Integrity and idempotent processingUse checksums and immutable source objects for file downloader library from frontend to backend deduplication and audit. Derive output keys from asset, profile, and transform version so retries cannot corrupt a completed rendition. Make completion callbacks and worker claims conditional on job version and attempt.
• Authorization at the edgeBind file downloader library from frontend to backend manifests and signed URLs to the viewer, entitlement version, and expiry. Propagate takedown, privacy, and subscription changes to edge caches with bounded revocation delay. Never let a cache hit bypass the policy decision for private or paid content.
• Cost, hot assets, and backpressureSeparate interactive manifest latency from expensive file downloader library from frontend to backend processing and encode work. Use queue priority, concurrency limits, and lifecycle policies for source and rendition storage. Measure cache hit rate, startup latency, processing backlog, failed bytes, and egress cost by profile.
• Process on upload versus on demandPrecompute common file downloader library from frontend to backend profiles and generate rare profiles on demand with a durable job state. Generating every possible profile up front wastes storage and processing budget.
• Origin storage versus CDN cachingKeep the origin authoritative and use CDN caching for immutable or versioned outputs with explicit invalidation. A cache cannot be the only copy of a file downloader library from frontend to backend rendition or the recovery path becomes undefined.
• Quality versus delivery costChoose profiles from device, bandwidth, and business requirements, then measure quality and egress by cohort. Maximal bitrate or resolution can make tail startup and cost unacceptable without improving viewing outcomes.