Design a Video Processing and Transcoding Pipeline — System Design Interview Practice

Interview prompt

Design upload, validation, transcoding, thumbnails, manifests, and CDN delivery so users can process an uploaded video reliably at scale.

• Define the source of truth for source object and job state and make retries idempotent.
• Use bounded, partitioned state to meet 1M uploads per day with multi-format outputs and upload acknowledgement <=1s.
• Separate the critical request path from transcoding, packaging, moderation, and CDN publish.
• Explain consistency, failure recovery, authorization, observability, and a degraded mode.

Requirements and scale assumptions

• Support the core workflow to process an uploaded video.
• Expose status, results, and freshness appropriate to upload, validation, transcoding, thumbnails, manifests, and CDN delivery.
• Support authorization, validation, updates, deletion, and recovery semantics.
• Meet upload acknowledgement <=1s under normal load.
• Scale to 1M uploads per day with multi-format outputs without a single hot key or unbounded synchronous work.
• Do not lose committed state; make retries and duplicate events safe.
• Degrade safely when downstream workers, caches, or external dependencies fail.
• 1M uploads per day with multi-format outputs
• Partition by the primary tenant, user, item, or geographic key and isolate hot partitions.
• Keep serving state bounded; retain raw events or durable records for replay and auditing.

• Peak scale: 1M uploads per day with multi-format outputs — Capacity assumption that drives partitioning and backpressure.
• Latency target: upload acknowledgement <=1s — User-facing budget for the primary request or read path.
• Durable boundary: Committed before async — The source of truth is source object and job state.
• Async boundary: At-least-once workers — Keep transcoding, packaging, moderation, and CDN publish off the synchronous path.

Key entities

• MediaAssetassetId, ownerId, sourceUri, checksum, privacy, status

  Canonical uploaded video transcoding pipeline asset and lifecycle state.

• MediaRenditionassetId, profile, codec, uri, checksum, status

  Derived video transcoding pipeline output identified by a deterministic profile and content hash.

• PlaybackSessionsessionId, assetId, viewerId, entitlementVersion, edgeRegion, expiresAt

  Short-lived video transcoding pipeline access session that binds authorization to delivery.

• ProcessingJobjobId, assetId, operation, attempt, checkpoint, status

  Retry-safe video transcoding pipeline processing job with checkpoints and per-rendition progress.

Data flow

1. 1. Reserve a resumable uploadThe video transcoding pipeline gateway authenticates the owner, reserves metadata, validates size and checksum, and returns a scoped upload URL.
2. 2. Commit and verify the sourceA completion callback verifies the video transcoding pipeline object, records an immutable checksum, and publishes a processing job only once.
3. 3. Process renditions asynchronouslyWorkers execute video transcoding pipeline transforms with deterministic profiles, checkpointing, bounded retries, and a dead-letter path for corrupt inputs.
4. 4. Publish an entitlement-aware manifestA manifest projection exposes only completed video transcoding pipeline renditions and carries policy, checksum, and freshness metadata.
5. 5. Deliver, invalidate, and recoverCDN delivery is protected by expiring URLs and revocation signals; failed video transcoding pipeline jobs and stale manifests are replayable without duplicating outputs.

Deep dives and trade-offs

• Integrity and idempotent processingUse checksums and immutable source objects for video transcoding pipeline deduplication and audit. Derive output keys from asset, profile, and transform version so retries cannot corrupt a completed rendition. Make completion callbacks and worker claims conditional on job version and attempt.
• Authorization at the edgeBind video transcoding pipeline manifests and signed URLs to the viewer, entitlement version, and expiry. Propagate takedown, privacy, and subscription changes to edge caches with bounded revocation delay. Never let a cache hit bypass the policy decision for private or paid content.
• Cost, hot assets, and backpressureSeparate interactive manifest latency from expensive video transcoding pipeline processing and encode work. Use queue priority, concurrency limits, and lifecycle policies for source and rendition storage. Measure cache hit rate, startup latency, processing backlog, failed bytes, and egress cost by profile.
• Process on upload versus on demandPrecompute common video transcoding pipeline profiles and generate rare profiles on demand with a durable job state. Generating every possible profile up front wastes storage and processing budget.
• Origin storage versus CDN cachingKeep the origin authoritative and use CDN caching for immutable or versioned outputs with explicit invalidation. A cache cannot be the only copy of a video transcoding pipeline rendition or the recovery path becomes undefined.
• Quality versus delivery costChoose profiles from device, bandwidth, and business requirements, then measure quality and egress by cohort. Maximal bitrate or resolution can make tail startup and cost unacceptable without improving viewing outcomes.