Design a Hybrid Cloud Infrastructure Solution — System Design Interview Practice

Interview prompt

Design a hybrid-cloud platform that manages on-premises and public-cloud resources with private connectivity, consistent identity and policy, unified observability, and safe workload placement.

• Separate desired resource and policy state from provider observations, local control planes, network routes, and workload data.
• Use adapters and asynchronous reconciliation so cloud API failure or a disconnected site cannot corrupt desired state.
• Enforce identity, encryption, residency, segmentation, and policy consistently while allowing provider-specific capabilities.
• Explain bandwidth constraints, split-brain connectivity, certificate rotation, disaster recovery, placement, and auditability.

Requirements and scale assumptions

• Register sites and cloud accounts, discover resources, apply declarative policies, provision services, and manage private routes.
• Expose desired versus observed state, connectivity, policy drift, capacity, health, and deployment operation progress.
• Support role-based access, secrets and certificate rotation, workload migration, backup, site isolation, and audit export.
• Target p95 control-plane status latency below 500 ms and keep application traffic within a site-specific network budget.
• Manage thousands of resources across cloud accounts and disconnected sites with site-scoped queues and policy workers.
• Make reconciliation, provisioning, policy evaluation, and certificate operations idempotent and auditable.
• Serve last-known state and queue changes during link loss while preventing unsafe cross-boundary mutations.
• Manage 50 sites, 20 cloud accounts, and 100,000 resources with intermittent links and regional policy differences.
• Partition desired and observed state by site, account, tenant, and provider; isolate disconnected or noisy sites.
• Retain encrypted policy decisions, resource snapshots, connectivity events, and audit records for replay.

• Managed resources: 100K resources — Resource scale drives discovery sharding, policy queues, and bounded status projections.
• Status latency: p95 <=500ms — Control-plane reads remain interactive even when a remote site is disconnected.
• Durable boundary: Committed before async — The source of truth is Manage on-premises and cloud resources; Deploy services across hybrid environment.
• Async boundary: At-least-once workers — Keep Azure Arc for hybrid management, Azure Stack for on-premises cloud, ExpressRoute for private connectivity off the synchronous path.

Key entities

• ResourceSpecresourceId, tenantId, desiredState, version, policyVersion, updatedAt

  Versioned desired state for a hybrid cloud infrastructure solution managed resource.

• OperationoperationId, resourceId, requestHash, step, attempt, status

  Durable hybrid cloud infrastructure solution reconciliation operation with per-step progress.

• PolicyVersionpolicyId, scope, version, rules, effectiveAt, status

  Auditable hybrid cloud infrastructure solution policy evaluated before provisioning or mutation.

• ReconciliationCheckpointresourceId, provider, observedVersion, cursor, lastError, updatedAt

  Provider-specific hybrid cloud infrastructure solution observation and recovery cursor.

Data flow

1. 1. Accept a desired-state commandThe hybrid cloud infrastructure solution control plane authenticates the tenant, validates policy and quotas, checks the expected version, and records the desired state.
2. 2. Plan a safe operationA planner turns hybrid cloud infrastructure solution desired state into ordered, bounded steps with dependency checks, blast-radius limits, and rollback metadata.
3. 3. Reconcile providers asynchronouslyWorkers apply hybrid cloud infrastructure solution operations through provider adapters, persist checkpoints, rate-limit calls, and treat unknown outcomes as observable state.
4. 4. Publish observed healthThe serving projection joins desired and observed hybrid cloud infrastructure solution state with operation status, policy version, freshness, and actionable errors.
5. 5. Recover and auditRetries, dead letters, drift detection, and operator approvals repair hybrid cloud infrastructure solution resources without losing the original command or provider evidence.

Deep dives and trade-offs

• Desired versus observed stateKeep hybrid cloud infrastructure solution desired state separate from provider-observed state and show both to operators. Make every reconciliation step conditional and resumable so a worker crash does not restart unsafe effects. Version policy and resource state so old operations cannot overwrite newer intent.
• Provider failures and unknown outcomesUse provider-specific idempotency tokens and query-after-timeout behavior for hybrid cloud infrastructure solution operations. Bound retries with exponential backoff, circuit breakers, and per-provider quotas. Route irreconcilable drift to an approval or quarantine path instead of retrying forever.
• Blast radius and operationsPartition hybrid cloud infrastructure solution work by tenant, region, cluster, or resource class and cap concurrent mutations. Audit who changed desired state, which policy allowed it, and what provider evidence was observed. Alert on drift age, operation backlog, failed steps, policy denials, and stale observations.
• Push versus pull reconciliationUse event triggers for fast response and periodic scans for missed events, drift, and recovery. A push-only hybrid cloud infrastructure solution controller silently misses changes when a provider event is lost.
• Central control plane versus provider-native controllersKeep policy, intent, and audit centralized while isolating provider-specific application logic behind adapters. A monolithic controller becomes hard to scale and couples unrelated provider failure domains.
• Automatic repair versus approvalAutomate low-risk, reversible hybrid cloud infrastructure solution changes and require approval for destructive or high-blast-radius operations. Full automation without policy or blast-radius controls can turn a transient signal into a widespread outage.