Design a Control Plane for a Distributed Database — System Design Interview Practice

Interview prompt

Design a control plane for a distributed database that provisions clusters, manages topology and upgrades, scales capacity, detects failures, and safely reconciles desired state with live replicas.

• Keep cluster desired state, observed topology, node leases, operation plans, and database data-plane consensus separate.
• Use a durable operation state machine and idempotent reconcilers for provisioning, failover, repair, resize, and upgrade.
• Preserve quorum and fencing invariants while coordinating cloud APIs, disks, networking, certificates, and backups.
• Explain tenant isolation, split brain, stuck operations, operator approval, rollback, and control-plane disaster recovery.

Requirements and scale assumptions

• Create clusters, add and remove replicas, change topology, rotate certificates, configure backups, and upgrade versions.
• Expose operation progress, quorum health, replica lag, capacity, drift, failure reason, and safe next actions.
• Support tenant-scoped roles, approvals for destructive operations, pause/resume, rollback, audit, and restore testing.
• Keep routine rolling operations within a bounded availability budget and never trade quorum safety for speed.
• Manage thousands of clusters and tenants with cluster-scoped queues, leases, rate limits, and isolated reconcilers.
• Make every provider action and topology transition idempotent, fenced, and recoverable after coordinator failure.
• Continue serving safe status reads while queuing mutations when a provider, node group, or control-plane dependency fails.
• Multi-tenancy support
• Partition by the primary tenant, user, item, or geographic key and isolate hot partitions.
• Keep serving state bounded; retain raw events or durable records for replay and auditing.

• Peak scale: Multi-tenancy support — Capacity assumption that drives partitioning and backpressure.
• Latency target: Minimize downtime during operations — User-facing budget for the primary request or read path.
• Durable boundary: Committed before async — The source of truth is Cluster provisioning and configuration; Automatic failover and recovery.
• Async boundary: At-least-once workers — Keep Consensus algorithms (Raft, Paxos), State machine replication, Leader election for coordination off the synchronous path.

Key entities

• SourcePartitionsourceId, partitionId, cursor, schemaVersion, watermark, status

  Replayable control plane source evidence and ingestion cursor.

• SchemaVersiondatasetId, version, compatibility, owner, effectiveAt, status

  Governed control plane contract used to validate producers and consumers.

• ProcessingRunrunId, inputWatermark, checkpoint, qualityStatus, codeVersion, status

  Checkpointed control plane processing attempt with quality and lineage metadata.

• AnalyticalDatasetdatasetId, partition, watermark, schemaVersion, qualityStatus, location

  Curated control plane serving partition with freshness and quality state.

Data flow

1. 1. Register sources and contractsThe control plane catalog records owners, schemas, compatibility rules, retention, lineage, and partitioning before data is accepted.
2. 2. Ingest with backpressureConnectors checkpoint control plane source cursors, validate schema and deduplication keys, and slow producers when downstream capacity is exhausted.
3. 3. Process event time with checkpointsStream or batch engines compute control plane transformations using watermarks, late-data policy, state checkpoints, and deterministic code versions.
4. 4. Publish quality-gated datasetsOnly control plane outputs that pass completeness, freshness, validity, and privacy checks become visible to analytical consumers.
5. 5. Serve, replay, and reconcileConsumers read bounded partitions with freshness metadata while operators replay failed control plane ranges and compare output checksums.

Deep dives and trade-offs

• Schema evolution and data qualityVersion control plane contracts and make compatibility rules explicit for every producer and consumer. Quarantine malformed partitions instead of poisoning the whole dataset. Track row counts, null rates, duplicates, distribution changes, and policy violations by partition.
• Watermarks, late data, and exactly-once effectsUse source cursors and event-time watermarks for control plane progress, not wall-clock assumptions. Make checkpoints, output keys, and sink commits retry-safe under at-least-once delivery. Document how late events revise windows, aggregates, or snapshots.
• Replay, lineage, and costKeep immutable control plane raw evidence and code or schema versions so failed outputs can be reproduced. Separate hot serving storage from cold retention and cap replay concurrency. Measure freshness, backlog, compute cost, storage growth, and quality-gate failure rate.
• Streaming versus batchUse streaming for freshness-critical control plane paths and batch for backfills, compaction, and expensive recomputation. Forcing every workload into streaming makes state, replay, and cost harder to operate.
• Raw retention versus curated-only storageRetain enough immutable raw evidence for replay, audit, and correction, then tier or expire it according to policy. Without raw evidence, a bad transformation can require an unreproducible emergency fix.
• Central warehouse versus domain-owned datasetsCentralize governance and discovery while letting domain owners own contracts and quality signals. A single team owning every transformation becomes a delivery bottleneck and hides data ownership.