Design a System to Monitor the Health of a Cluster — System Design Interview Practice

Interview prompt

Design a cluster-health monitoring system that discovers nodes, collects heartbeats and resource signals, detects quorum or capacity risks, and serves reliable operator views.

• Define node identity, heartbeats, leases, metric labels, health states, clock/skew handling, cluster membership, and quorum semantics.
• Separate liveness from performance and application health; handle partitions, flapping nodes, stale heartbeats, autoscaling, and cardinality limits.
• Keep node agents lightweight and stream telemetry into bounded aggregates, with durable snapshots for incident investigation.
• Explain alert hysteresis, regional failure, secure enrollment, retention, operator access, observability, and degraded status reads.

Requirements and scale assumptions

• Enroll and discover nodes, receive heartbeats and CPU/memory/disk/network metrics, and compute node and cluster health.
• Expose current status, historical trends, capacity summaries, quorum warnings, topology, and evidence behind each alert.
• Support node replacement, maintenance suppression, secure agent rotation, alert acknowledgment, replay, and recovery after collector loss.
• Reflect heartbeat loss within 30 seconds and serve p95 health queries under 300ms.
• Scale to 10k nodes per cluster and frequent autoscaling without a single hot key or unbounded synchronous work.
• Do not lose committed state; make retries and duplicate events safe.
• Degrade safely when downstream workers, caches, or external dependencies fail.
• 10k nodes per cluster sending heartbeats every 10 seconds
• Partition by the primary tenant, user, item, or geographic key and isolate hot partitions.
• Keep serving state bounded; retain raw events or durable records for replay and auditing.

• Peak scale: 10k nodes/cluster; 1k clusters — Capacity assumption that drives partitioning and backpressure.
• Latency target: stale detection < 30s; queries < 300ms — User-facing budget for the primary request or read path.
• Durable boundary: Committed before async — Signed heartbeats and metric samples are authoritative observations; health summaries are derived.
• Async boundary: At-least-once workers — Keep Heartbeat mechanism for liveness, Time-series database for metrics, Gossip protocol for distributed health off the synchronous path.

Key entities

• ResourceSpecresourceId, tenantId, desiredState, version, policyVersion, updatedAt

  Versioned desired state for a system to monitor the health of a cluster managed resource.

• OperationoperationId, resourceId, requestHash, step, attempt, status

  Durable system to monitor the health of a cluster reconciliation operation with per-step progress.

• PolicyVersionpolicyId, scope, version, rules, effectiveAt, status

  Auditable system to monitor the health of a cluster policy evaluated before provisioning or mutation.

• ReconciliationCheckpointresourceId, provider, observedVersion, cursor, lastError, updatedAt

  Provider-specific system to monitor the health of a cluster observation and recovery cursor.

Data flow

1. 1. Accept a desired-state commandThe system to monitor the health of a cluster control plane authenticates the tenant, validates policy and quotas, checks the expected version, and records the desired state.
2. 2. Plan a safe operationA planner turns system to monitor the health of a cluster desired state into ordered, bounded steps with dependency checks, blast-radius limits, and rollback metadata.
3. 3. Reconcile providers asynchronouslyWorkers apply system to monitor the health of a cluster operations through provider adapters, persist checkpoints, rate-limit calls, and treat unknown outcomes as observable state.
4. 4. Publish observed healthThe serving projection joins desired and observed system to monitor the health of a cluster state with operation status, policy version, freshness, and actionable errors.
5. 5. Recover and auditRetries, dead letters, drift detection, and operator approvals repair system to monitor the health of a cluster resources without losing the original command or provider evidence.

Deep dives and trade-offs

• Desired versus observed stateKeep system to monitor the health of a cluster desired state separate from provider-observed state and show both to operators. Make every reconciliation step conditional and resumable so a worker crash does not restart unsafe effects. Version policy and resource state so old operations cannot overwrite newer intent.
• Provider failures and unknown outcomesUse provider-specific idempotency tokens and query-after-timeout behavior for system to monitor the health of a cluster operations. Bound retries with exponential backoff, circuit breakers, and per-provider quotas. Route irreconcilable drift to an approval or quarantine path instead of retrying forever.
• Blast radius and operationsPartition system to monitor the health of a cluster work by tenant, region, cluster, or resource class and cap concurrent mutations. Audit who changed desired state, which policy allowed it, and what provider evidence was observed. Alert on drift age, operation backlog, failed steps, policy denials, and stale observations.
• Push versus pull reconciliationUse event triggers for fast response and periodic scans for missed events, drift, and recovery. A push-only system to monitor the health of a cluster controller silently misses changes when a provider event is lost.
• Central control plane versus provider-native controllersKeep policy, intent, and audit centralized while isolating provider-specific application logic behind adapters. A monolithic controller becomes hard to scale and couples unrelated provider failure domains.
• Automatic repair versus approvalAutomate low-risk, reversible system to monitor the health of a cluster changes and require approval for destructive or high-blast-radius operations. Full automation without policy or blast-radius controls can turn a transient signal into a widespread outage.