Design a GPU Cluster Management System for ML Training — System Design Interview Practice

Interview prompt

Design a GPU cluster manager for ML training that schedules distributed jobs, allocates scarce accelerator and network resources across tenants, supports preemption and checkpoint recovery, and optimizes utilization and cost.

• Model job requirements, GPU type/count, topology, checkpoints, priority, quota, reservation, and tenant fairness explicitly.
• Schedule gang jobs with topology awareness for NVLink/NVSwitch, detect unhealthy devices, and reclaim capacity safely.
• Separate desired job state from node observations, utilization telemetry, billing, and checkpoint storage.
• Explain preemption policy, starvation, elastic training, node loss, confidential data, quotas, and cost attribution.

Requirements and scale assumptions

• Submit training jobs, validate resource requests, queue by priority and fair share, allocate GPUs, and launch distributed workers.
• Expose queue wait, placement, GPU health, utilization, preemption, checkpoint, restart, completion, and cost state.
• Support tenant quotas, reservations, cancellation, checkpoint restore, chargeback, node drain, and administrative audit.
• Target above 80% aggregate GPU utilization without violating latency, fairness, or thermal and power limits.
• Manage 10,000 GPUs and thousands of concurrent jobs with scheduler shards and per-tenant admission control.
• Make allocation, preemption, checkpoint, and billing events idempotent; never allocate one GPU to two jobs.
• Drain unhealthy nodes, restore from checkpoints, and preserve queued intent when scheduler or telemetry services fail.
• Operate 10,000 GPUs across accelerator generations, regions, and network topologies.
• Partition scheduling by cluster, tenant, queue, and accelerator type; isolate large distributed jobs.
• Retain job specifications, allocation decisions, device health, checkpoints, utilization samples, and billing evidence.

• Peak scale: Provide cost allocation — Capacity assumption that drives partitioning and backpressure.
• Latency target: GPU utilization above 80% cluster-wide — User-facing budget for the primary request or read path.
• Durable boundary: Committed before async — The source of truth is Schedule training jobs across GPU nodes; Implement fair-share resource allocation.
• Async boundary: At-least-once workers — Keep Use Kubernetes with GPU operator for orchestration, Implement Volcano or Kueue for batch scheduling, Use topology-aware scheduling for NVLink/NVSwitch off the synchronous path.

Key entities

• DatasetVersiondatasetId, version, schemaHash, qualityStatus, lineage, createdAt

  Immutable gpu cluster management system input version used for reproducible training, evaluation, or replay.

• FeatureSnapshotentityId, featureSetVersion, eventTime, values, sourceWatermarks

  Point-in-time gpu cluster management system features with source watermarks so online and offline values can be compared.

• TrainingRunrunId, datasetVersion, codeVersion, metrics, artifactUri, status

  Audited gpu cluster management system run that records data, code, dependency, and evaluation lineage.

• ModelVersionmodelId, version, stage, schema, qualityGates, endpoint

  A promotable gpu cluster management system model version with rollout state, contract, and rollback metadata.

Data flow

1. 1. Register and validate training dataThe gpu cluster management system gateway records an immutable dataset version, schema, lineage, quality status, and privacy disposition.
2. 2. Build point-in-time featuresFeature workers join gpu cluster management system inputs using event-time watermarks, prevent leakage, and publish the same feature contract for training and serving.
3. 3. Train and evaluate asynchronouslyThe orchestrator schedules gpu cluster management system runs with checkpointed artifacts, reproducible environments, and metrics tied to the exact input versions.
4. 4. Gate and serve a model versionA registry compares gpu cluster management system quality, bias, safety, and compatibility gates before canary or production rollout with an immediate rollback pointer.
5. 5. Monitor drift and learn from feedbackOnline inference records latency, errors, drift, and delayed labels so gpu cluster management system retraining is evidence-driven rather than triggered by guesswork.

Deep dives and trade-offs

• Reproducibility and leakage preventionPin gpu cluster management system data, feature, code, dependency, and model versions for every run. Use point-in-time joins and quarantine failed quality or privacy checks before training. Keep raw inputs and artifacts immutable so a result can be replayed after a dependency changes.
• Safe promotion and serving contractsSeparate gpu cluster management system model registration from deployment and require signed artifacts plus schema compatibility. Use shadow traffic, canaries, rollback pointers, and per-version latency/error budgets. Return model version and feature freshness so clients can explain or reproduce a prediction.
• Drift, feedback, and costMeasure feature drift, prediction drift, label delay, and segment-level quality for gpu cluster management system rather than only aggregate accuracy. Sample expensive inference and cap retraining concurrency with an explicit GPU or compute budget. Keep human corrections and delayed labels linked to the original prediction and model version.
• Batch versus online featuresPrefer a shared feature contract with batch backfills and a low-latency online serving path for decisions that need freshness. Two independently defined transformations create training-serving skew and hard-to-debug regressions.
• Synchronous versus asynchronous inferenceKeep interactive gpu cluster management system inference synchronous within a strict budget and queue large or expensive jobs. A request path that waits for model loading, enrichment, or retraining turns downstream slowness into an outage.
• Global model versus segment modelsStart with one versioned model and add segment-specific models only when quality or policy evidence justifies the operational cost. Many simultaneously active versions multiply monitoring, rollback, and data-lineage burden.