Build a Multi-Cloud Kubernetes Orchestration Platform — System Design Interview Practice

Interview prompt

Design a multi-cloud Kubernetes control plane that deploys workloads consistently across providers, centralizes policy and observability, and recovers workloads after a cloud or cluster failure.

• Keep desired application state in a versioned Git or control-plane store while treating each provider cluster as an eventually reconciled target.
• Define portable workload contracts, provider-specific adapters, placement constraints, secrets handling, and policy inheritance.
• Make reconciliation idempotent, observable, rate-limited, and safe when a cluster is unreachable or partially applied.
• Explain fleet upgrades, cross-cloud networking, identity, cost, disaster recovery, and split-brain prevention.

Requirements and scale assumptions

• Register clusters, credentials, capabilities, and health; onboard workloads through declarative manifests or GitOps.
• Schedule or target releases across clusters, report reconciliation status, policy violations, health, and drift.
• Apply access policies, rotate secrets, upgrade clusters, drain failed nodes, and restore workloads from a known version.
• Manage 100+ clusters across at least three providers while keeping control-plane APIs responsive during fleet changes.
• Reconcile clusters independently so one provider outage cannot block healthy clusters or overload the control plane.
• Never lose desired state or audit history; make apply, retry, upgrade, and rollback operations idempotent.
• Degrade by serving the last observed state and queueing reconciliation when a target cluster or provider API fails.
• Operate 100+ clusters, 10,000 namespaces, and tens of thousands of workload objects with bursty release waves.
• Partition reconciliation by cluster and tenant, isolate slow provider APIs, and cap concurrent operations per target.
• Keep desired state, observed snapshots, policy decisions, and audit events durable; cache only bounded status views.

• Fleet size: 100+ clusters — Drives cluster-scoped queues, provider rate limits, and reconciliation isolation.
• Control-plane status: p95 <=500ms — Status reads stay interactive while deployments and reconciliation continue asynchronously.
• Durable boundary: Committed before async — The source of truth is versioned desired state; cluster observations are reconciled projections.
• Async boundary: At-least-once workers — Keep provider reconciliation, upgrades, policy scans, metrics aggregation, and backups asynchronous.

Key entities

• ResourceSpecresourceId, tenantId, desiredState, version, policyVersion, updatedAt

  Versioned desired state for a multi cloud kubernetes orchestration platform managed resource.

• OperationoperationId, resourceId, requestHash, step, attempt, status

  Durable multi cloud kubernetes orchestration platform reconciliation operation with per-step progress.

• PolicyVersionpolicyId, scope, version, rules, effectiveAt, status

  Auditable multi cloud kubernetes orchestration platform policy evaluated before provisioning or mutation.

• ReconciliationCheckpointresourceId, provider, observedVersion, cursor, lastError, updatedAt

  Provider-specific multi cloud kubernetes orchestration platform observation and recovery cursor.

Data flow

1. 1. Accept a desired-state commandThe multi cloud kubernetes orchestration platform control plane authenticates the tenant, validates policy and quotas, checks the expected version, and records the desired state.
2. 2. Plan a safe operationA planner turns multi cloud kubernetes orchestration platform desired state into ordered, bounded steps with dependency checks, blast-radius limits, and rollback metadata.
3. 3. Reconcile providers asynchronouslyWorkers apply multi cloud kubernetes orchestration platform operations through provider adapters, persist checkpoints, rate-limit calls, and treat unknown outcomes as observable state.
4. 4. Publish observed healthThe serving projection joins desired and observed multi cloud kubernetes orchestration platform state with operation status, policy version, freshness, and actionable errors.
5. 5. Recover and auditRetries, dead letters, drift detection, and operator approvals repair multi cloud kubernetes orchestration platform resources without losing the original command or provider evidence.

Deep dives and trade-offs

• Desired versus observed stateKeep multi cloud kubernetes orchestration platform desired state separate from provider-observed state and show both to operators. Make every reconciliation step conditional and resumable so a worker crash does not restart unsafe effects. Version policy and resource state so old operations cannot overwrite newer intent.
• Provider failures and unknown outcomesUse provider-specific idempotency tokens and query-after-timeout behavior for multi cloud kubernetes orchestration platform operations. Bound retries with exponential backoff, circuit breakers, and per-provider quotas. Route irreconcilable drift to an approval or quarantine path instead of retrying forever.
• Blast radius and operationsPartition multi cloud kubernetes orchestration platform work by tenant, region, cluster, or resource class and cap concurrent mutations. Audit who changed desired state, which policy allowed it, and what provider evidence was observed. Alert on drift age, operation backlog, failed steps, policy denials, and stale observations.
• Push versus pull reconciliationUse event triggers for fast response and periodic scans for missed events, drift, and recovery. A push-only multi cloud kubernetes orchestration platform controller silently misses changes when a provider event is lost.
• Central control plane versus provider-native controllersKeep policy, intent, and audit centralized while isolating provider-specific application logic behind adapters. A monolithic controller becomes hard to scale and couples unrelated provider failure domains.
• Automatic repair versus approvalAutomate low-risk, reversible multi cloud kubernetes orchestration platform changes and require approval for destructive or high-blast-radius operations. Full automation without policy or blast-radius controls can turn a transient signal into a widespread outage.