Design a High-Performance Computing Cluster — System Design Interview Practice

Interview prompt

Design an HPC cluster for tightly coupled scientific workloads with low-latency interconnects, parallel storage, topology-aware scheduling, checkpointing, and fair multi-tenant batch execution.

• Model jobs, node groups, topology, reservations, queues, parallel filesets, checkpoints, and tenant quotas explicitly.
• Place MPI ranks and GPUs with NUMA, fabric, and storage locality while preventing one large job from starving the queue.
• Checkpoint long simulations, drain failed nodes safely, and keep scheduler metadata separate from high-throughput data paths.
• Explain fabric health, stragglers, preemption, accounting, isolation, reproducibility, and recovery after controller failure.

Requirements and scale assumptions

• Provision compute and storage, submit MPI or batch jobs, select queues and reservations, and stage input and output datasets.
• Expose queue wait, placement, fabric health, node state, job progress, checkpoint age, utilization, and failure reason.
• Support fair-share quotas, cancellation, node drain, checkpoint restore, accounting, software environments, and audit history.
• Keep inter-node fabric latency below two microseconds where the hardware supports it and keep scheduler decisions bounded.
• Scale to thousands of nodes and concurrent jobs with queue and topology partitions.
• Make allocation, checkpoint, stage, and accounting transitions idempotent and never double-allocate a node.
• Continue safe status reads and preserve queued intent when a scheduler, filesystem, or node group is degraded.
• Operate 5,000 CPU/GPU nodes, 100 PB of parallel storage, and thousands of jobs with bursty reservations.
• Partition queues by tenant, project, resource shape, and topology; isolate large MPI jobs.
• Retain job specifications, placements, checkpoints, node health, and accounting records for replay and audit.

• Cluster scale: 5K nodes — Node and topology scale drives scheduler sharding, reservations, and fabric monitoring.
• Fabric latency: <2 microseconds — Tightly coupled jobs depend on low-latency interconnects; scheduler latency is a separate budget.
• Durable boundary: Committed before async — The source of truth is Deploy large-scale compute clusters; Run parallel workloads efficiently.
• Async boundary: At-least-once workers — Keep Azure CycleCloud for cluster management, Azure Batch for job scheduling, HPC Cache for file system caching off the synchronous path.

Key entities

• ResourceSpecresourceId, tenantId, desiredState, version, policyVersion, updatedAt

  Versioned desired state for a high performance computing cluster managed resource.

• OperationoperationId, resourceId, requestHash, step, attempt, status

  Durable high performance computing cluster reconciliation operation with per-step progress.

• PolicyVersionpolicyId, scope, version, rules, effectiveAt, status

  Auditable high performance computing cluster policy evaluated before provisioning or mutation.

• ReconciliationCheckpointresourceId, provider, observedVersion, cursor, lastError, updatedAt

  Provider-specific high performance computing cluster observation and recovery cursor.

Data flow

1. 1. Accept a desired-state commandThe high performance computing cluster control plane authenticates the tenant, validates policy and quotas, checks the expected version, and records the desired state.
2. 2. Plan a safe operationA planner turns high performance computing cluster desired state into ordered, bounded steps with dependency checks, blast-radius limits, and rollback metadata.
3. 3. Reconcile providers asynchronouslyWorkers apply high performance computing cluster operations through provider adapters, persist checkpoints, rate-limit calls, and treat unknown outcomes as observable state.
4. 4. Publish observed healthThe serving projection joins desired and observed high performance computing cluster state with operation status, policy version, freshness, and actionable errors.
5. 5. Recover and auditRetries, dead letters, drift detection, and operator approvals repair high performance computing cluster resources without losing the original command or provider evidence.

Deep dives and trade-offs

• Desired versus observed stateKeep high performance computing cluster desired state separate from provider-observed state and show both to operators. Make every reconciliation step conditional and resumable so a worker crash does not restart unsafe effects. Version policy and resource state so old operations cannot overwrite newer intent.
• Provider failures and unknown outcomesUse provider-specific idempotency tokens and query-after-timeout behavior for high performance computing cluster operations. Bound retries with exponential backoff, circuit breakers, and per-provider quotas. Route irreconcilable drift to an approval or quarantine path instead of retrying forever.
• Blast radius and operationsPartition high performance computing cluster work by tenant, region, cluster, or resource class and cap concurrent mutations. Audit who changed desired state, which policy allowed it, and what provider evidence was observed. Alert on drift age, operation backlog, failed steps, policy denials, and stale observations.
• Push versus pull reconciliationUse event triggers for fast response and periodic scans for missed events, drift, and recovery. A push-only high performance computing cluster controller silently misses changes when a provider event is lost.
• Central control plane versus provider-native controllersKeep policy, intent, and audit centralized while isolating provider-specific application logic behind adapters. A monolithic controller becomes hard to scale and couples unrelated provider failure domains.
• Automatic repair versus approvalAutomate low-risk, reversible high performance computing cluster changes and require approval for destructive or high-blast-radius operations. Full automation without policy or blast-radius controls can turn a transient signal into a widespread outage.