Design a Complete CI/CD Pipeline — System Design Interview Practice

Interview prompt

Design a CI/CD platform that turns source changes into tested, signed artifacts and safely deploys them across environments with approvals, rollback, and auditable provenance.

• Separate immutable source revisions and signed artifacts from mutable run state, environment configuration, and deployment status.
• Use queue-backed isolated workers with cacheable build inputs, concurrency limits, and per-repository fairness.
• Require test, security, provenance, and approval gates before promotion, with an atomic environment release pointer.
• Explain secrets isolation, untrusted pull-request execution, retries, rollback, artifact retention, and audit trails.

Requirements and scale assumptions

• Register repositories and pipeline definitions, trigger runs, schedule build and test stages, and store logs and artifacts.
• Promote signed artifacts through environments with approvals, progressive rollout, health checks, rollback, and release history.
• Expose run status, stage logs, cache hits, test results, security findings, deployment events, and cancellation.
• Meet queue acknowledgement <=1s under normal load.
• Scale to 100K pipeline runs per day with bursty repositories without a single hot key or unbounded synchronous work.
• Do not lose committed state; make retries and duplicate events safe.
• Degrade safely when downstream workers, caches, or external dependencies fail.
• 100K pipeline runs per day with bursty repositories
• Partition by the primary tenant, user, item, or geographic key and isolate hot partitions.
• Keep serving state bounded; retain raw events or durable records for replay and auditing.

• Peak scale: 100K pipeline runs per day with bursty repositories — Capacity assumption that drives partitioning and backpressure.
• Latency target: queue acknowledgement <=1s — User-facing budget for the primary request or read path.
• Durable boundary: Committed before async — The source of truth is pipeline definitions and execution state.
• Async boundary: At-least-once workers — Keep workers, artifact storage, deployment, and notifications off the synchronous path.

Key entities

• SourcePartitionsourceId, partitionId, cursor, schemaVersion, watermark, status

  Replayable ci cd pipeline source evidence and ingestion cursor.

• SchemaVersiondatasetId, version, compatibility, owner, effectiveAt, status

  Governed ci cd pipeline contract used to validate producers and consumers.

• ProcessingRunrunId, inputWatermark, checkpoint, qualityStatus, codeVersion, status

  Checkpointed ci cd pipeline processing attempt with quality and lineage metadata.

• AnalyticalDatasetdatasetId, partition, watermark, schemaVersion, qualityStatus, location

  Curated ci cd pipeline serving partition with freshness and quality state.

Data flow

1. 1. Register sources and contractsThe ci cd pipeline catalog records owners, schemas, compatibility rules, retention, lineage, and partitioning before data is accepted.
2. 2. Ingest with backpressureConnectors checkpoint ci cd pipeline source cursors, validate schema and deduplication keys, and slow producers when downstream capacity is exhausted.
3. 3. Process event time with checkpointsStream or batch engines compute ci cd pipeline transformations using watermarks, late-data policy, state checkpoints, and deterministic code versions.
4. 4. Publish quality-gated datasetsOnly ci cd pipeline outputs that pass completeness, freshness, validity, and privacy checks become visible to analytical consumers.
5. 5. Serve, replay, and reconcileConsumers read bounded partitions with freshness metadata while operators replay failed ci cd pipeline ranges and compare output checksums.

Deep dives and trade-offs

• Schema evolution and data qualityVersion ci cd pipeline contracts and make compatibility rules explicit for every producer and consumer. Quarantine malformed partitions instead of poisoning the whole dataset. Track row counts, null rates, duplicates, distribution changes, and policy violations by partition.
• Watermarks, late data, and exactly-once effectsUse source cursors and event-time watermarks for ci cd pipeline progress, not wall-clock assumptions. Make checkpoints, output keys, and sink commits retry-safe under at-least-once delivery. Document how late events revise windows, aggregates, or snapshots.
• Replay, lineage, and costKeep immutable ci cd pipeline raw evidence and code or schema versions so failed outputs can be reproduced. Separate hot serving storage from cold retention and cap replay concurrency. Measure freshness, backlog, compute cost, storage growth, and quality-gate failure rate.
• Streaming versus batchUse streaming for freshness-critical ci cd pipeline paths and batch for backfills, compaction, and expensive recomputation. Forcing every workload into streaming makes state, replay, and cost harder to operate.
• Raw retention versus curated-only storageRetain enough immutable raw evidence for replay, audit, and correction, then tier or expire it according to policy. Without raw evidence, a bad transformation can require an unreproducible emergency fix.
• Central warehouse versus domain-owned datasetsCentralize governance and discovery while letting domain owners own contracts and quality signals. A single team owning every transformation becomes a delivery bottleneck and hides data ownership.