Design an Automated Chaos Engineering Platform — System Design Interview Practice

Interview prompt

Design a controlled chaos-engineering platform that schedules approved fault experiments, limits blast radius, measures resilience hypotheses, and automatically stops or rolls back unsafe tests.

• Define experiment hypotheses, target selectors, fault types, blast-radius budgets, guardrails, stop conditions, approvals, and evidence.
• Require isolated environments or explicit production consent, progressive canaries, health gates, leases, kill switches, and automatic cleanup.
• Separate experiment control from fault agents, telemetry, analysis, and CI/CD gates; make runs idempotent and reproducible.
• Explain customer-impact detection, agent failure, network partitions, secrets, auditability, observability, and manual-only fallback.

Requirements and scale assumptions

• Create hypotheses and experiments, approve targets, inject bounded faults, observe health, stop/rollback, and publish resilience results.
• Support scheduled/CI runs, canary progression, network/CPU/disk/service faults, maintenance windows, owner notifications, and run comparison.
• Provide signed agent enrollment, kill switch, immutable evidence, teardown verification, experiment deletion, and recovery after controller failure.
• Enforce blast-radius and health guardrails so approved experiments stop before breaching customer-impact budgets.
• Run thousands of experiments across fleets and CI pipelines without a single hot key or unbounded synchronous work.
• Do not lose committed state; make retries and duplicate events safe.
• Degrade safely when downstream workers, caches, or external dependencies fail.
• 10k services, 1k concurrent experiments, and 100 CI pipelines
• Partition by the primary tenant, user, item, or geographic key and isolate hot partitions.
• Keep serving state bounded; retain raw events or durable records for replay and auditing.

• Peak scale: 1k concurrent experiments; 10k services — Capacity assumption that drives partitioning and backpressure.
• Latency target: guardrail stop < 60s; evidence complete — User-facing budget for the primary request or read path.
• Durable boundary: Committed before async — Approved experiment definitions, agent leases, and signed run evidence are authoritative.
• Async boundary: At-least-once workers — Keep Use Chaos Monkey, Litmus, or Gremlin as tooling base, Implement blast radius controls with feature flags, Use ML to identify untested failure modes off the synchronous path.

Key entities

• ResourceSpecresourceId, tenantId, desiredState, version, policyVersion, updatedAt

  Versioned desired state for a automated chaos engineering platform managed resource.

• OperationoperationId, resourceId, requestHash, step, attempt, status

  Durable automated chaos engineering platform reconciliation operation with per-step progress.

• PolicyVersionpolicyId, scope, version, rules, effectiveAt, status

  Auditable automated chaos engineering platform policy evaluated before provisioning or mutation.

• ReconciliationCheckpointresourceId, provider, observedVersion, cursor, lastError, updatedAt

  Provider-specific automated chaos engineering platform observation and recovery cursor.

Data flow

1. 1. Accept a desired-state commandThe automated chaos engineering platform control plane authenticates the tenant, validates policy and quotas, checks the expected version, and records the desired state.
2. 2. Plan a safe operationA planner turns automated chaos engineering platform desired state into ordered, bounded steps with dependency checks, blast-radius limits, and rollback metadata.
3. 3. Reconcile providers asynchronouslyWorkers apply automated chaos engineering platform operations through provider adapters, persist checkpoints, rate-limit calls, and treat unknown outcomes as observable state.
4. 4. Publish observed healthThe serving projection joins desired and observed automated chaos engineering platform state with operation status, policy version, freshness, and actionable errors.
5. 5. Recover and auditRetries, dead letters, drift detection, and operator approvals repair automated chaos engineering platform resources without losing the original command or provider evidence.

Deep dives and trade-offs

• Desired versus observed stateKeep automated chaos engineering platform desired state separate from provider-observed state and show both to operators. Make every reconciliation step conditional and resumable so a worker crash does not restart unsafe effects. Version policy and resource state so old operations cannot overwrite newer intent.
• Provider failures and unknown outcomesUse provider-specific idempotency tokens and query-after-timeout behavior for automated chaos engineering platform operations. Bound retries with exponential backoff, circuit breakers, and per-provider quotas. Route irreconcilable drift to an approval or quarantine path instead of retrying forever.
• Blast radius and operationsPartition automated chaos engineering platform work by tenant, region, cluster, or resource class and cap concurrent mutations. Audit who changed desired state, which policy allowed it, and what provider evidence was observed. Alert on drift age, operation backlog, failed steps, policy denials, and stale observations.
• Push versus pull reconciliationUse event triggers for fast response and periodic scans for missed events, drift, and recovery. A push-only automated chaos engineering platform controller silently misses changes when a provider event is lost.
• Central control plane versus provider-native controllersKeep policy, intent, and audit centralized while isolating provider-specific application logic behind adapters. A monolithic controller becomes hard to scale and couples unrelated provider failure domains.
• Automatic repair versus approvalAutomate low-risk, reversible automated chaos engineering platform changes and require approval for destructive or high-blast-radius operations. Full automation without policy or blast-radius controls can turn a transient signal into a widespread outage.