Design an Intelligent Capacity Planning System — System Design Interview Practice
Design an AI-driven capacity planning system that forecasts future resource needs, models growth scenarios, optimizes cloud spending, and provides procurement recommendations. Work through the requirements, architecture trade-offs, and an interactive design review.
Concepts and architecture decisions to consider
- aiopsConcept to explore
- capacity planningConcept to explore
- forecastingConcept to explore
- cost optimizationConcept to explore
- resource managementConcept to explore
- cloudConcept to explore
Interview prompt
Design an intelligent capacity-planning platform that forecasts resource demand, models growth and failure scenarios, alerts on saturation risk, and turns recommendations into governed infrastructure decisions.
- Define resource identity, utilization/availability signals, service dependencies, forecast horizon, scenarios, constraints, confidence, and ownership.
- Handle seasonality, growth changes, saturation, reserved capacity, quotas, outages, and uncertainty without presenting estimates as guarantees.
- Separate telemetry ingestion from forecasting, simulation, recommendation approval, procurement/workflow execution, and dashboard queries.
- Explain feedback, overrides, stale data, multi-cloud costs, auditability, observability, and last-approved-plan fallback.
Requirements and scale assumptions
- Collect resource metrics and demand drivers, generate baseline/scenario forecasts, calculate headroom, and flag future capacity risks.
- Provide forecasts with intervals, assumptions, cost/options, dependency impact, owner approval, and procurement or scaling recommendations.
- Support overrides, what-if simulations, data corrections, forecast versioning, alerts, recommendation execution, and rollback/audit.
- Publish a 90-day forecast with error and uncertainty by resource class and alert before agreed saturation thresholds are reached.
- Forecast millions of resource series across clouds without a single hot key or unbounded synchronous work.
- Do not lose committed state; make retries and duplicate events safe.
- Degrade safely when downstream workers, caches, or external dependencies fail.
- 5M resource series, 90-day horizons, and 10k service owners
- Partition by the primary tenant, user, item, or geographic key and isolate hot partitions.
- Keep serving state bounded; retain raw events or durable records for replay and auditing.
- Peak scale: 5M series; 10k service owners — Capacity assumption that drives partitioning and backpressure.
- Latency target: forecast intervals published; alerts before threshold — User-facing budget for the primary request or read path.
- Durable boundary: Committed before async — Versioned telemetry snapshots, forecasts, constraints, and approved plans are authoritative.
- Async boundary: At-least-once workers — Keep Use time-series decomposition for trend/seasonal analysis, Implement simulation-based capacity modeling, Use optimization algorithms for instance type selection off the synchronous path.
Key entities
- DemandSignalsourceId, resourceId, metric, eventTime, value, quality
Timestamped demand or utilization evidence for intelligent capacity planning system.
- CapacityForecastforecastId, resourceId, horizon, quantiles, drivers, modelVersion
Versioned forecast with intervals, assumptions, and confidence.
- PlanningScenarioscenarioId, baselineId, changes, dependencyImpact, costEstimate, createdBy
Auditable what-if capacity or failure scenario.
- CapacityDecisiondecisionId, scenarioId, recommendation, owner, approval, executionStatus
Governed capacity recommendation and execution state.
Data flow
- 1. Collect resource and demand evidenceThe intelligent capacity planning system gateway validates metric ownership, event time, resource identity, quality, and correction semantics.
- 2. Build baseline and scenario forecastsForecast workers combine demand drivers, seasonality, deployments, failures, and cost assumptions into versioned outputs.
- 3. Calculate headroom and riskPlanning workers estimate saturation horizon, dependency impact, confidence intervals, procurement options, and cost.
- 4. Govern a capacity decisionRecommendations carry evidence, owner, policy, approval, execution, and rollback state instead of mutating infrastructure directly.
- 5. Measure outcomes and correctOperators compare intelligent capacity planning system forecasts with actual demand, record overrides, replay bad ranges, and improve model calibration.
Deep dives and trade-offs
- Forecast quality and uncertaintyKeep baseline, interval, driver, horizon, and model versions together so planners understand uncertainty. Separate missing data from low demand and lower confidence when telemetry or dependencies are incomplete. Evaluate error by resource class, seasonality, scale event, and forecast horizon.
- Scenarios and dependency impactModel shared quotas, bottlenecks, failure domains, and correlated demand rather than forecasting each resource independently. Run what-if scenarios against an immutable baseline and never overwrite the production forecast. Show cost, headroom, assumptions, and downstream impact with every recommendation.
- Governed executionRequire owner, policy, blast-radius, approval, and rollback metadata before an action can execute. Keep recommendation state separate from observed infrastructure state and reconcile after execution. Measure approval latency, action success, cost variance, rollback, and forecast-to-outcome error.
- Baseline model versus scenario simulationUse a stable baseline for planning and isolate scenarios for what-if decisions and failure analysis. Mixing hypothetical demand into the baseline makes actual forecast accuracy impossible to measure.
- Automatic scaling versus approved capacityLet existing autoscaling handle bounded real-time response and use governed recommendations for procurement or structural changes. A forecast error should not trigger an unbounded or irreversible capacity action.
- Central forecast versus owner-specific modelsCentralize evidence, contracts, and evaluation while allowing resource owners to supply domain drivers and constraints. A single global model can hide resource-specific seasonality and policy limits.