Design a Predictive Auto-Scaling System — System Design Interview Practice

Interview prompt

Design a predictive autoscaling system that forecasts service demand, combines predictions with safety limits and current capacity, and proactively scales infrastructure without causing oscillation or runaway cost.

• Separate observed metrics, forecasts, scaling policy, capacity reservations, actuator commands, and resulting health signals.
• Forecast multiple horizons with confidence bands, combine prediction with reactive safeguards, and smooth actions with cooldowns.
• Coordinate dependent services in an order that preserves capacity and avoid scaling on corrupted, delayed, or feedback-loop data.
• Explain cold start, event-driven spikes, model drift, quota failure, rollback, cost budgets, and manual override.

Requirements and scale assumptions

• Collect service metrics and calendars, produce forecasts, evaluate confidence, and calculate a bounded desired capacity.
• Apply scale-up and scale-down actions across dependent services with approvals, cooldowns, health checks, and audit history.
• Expose forecast, recommendation, actual capacity, action result, cost, model version, and manual override state.
• Target demand forecast error within 15% on declared workloads while reactive utilization limits protect the SLO.
• Scale decisions for thousands of services with dependency-aware queues and per-service rate and cost limits.
• Make forecast publication and actuator commands idempotent and require observed state before repeating a scale action.
• Fall back to reactive autoscaling and safe capacity floors when forecasts, metrics, or provider APIs fail.
• Manage 10,000 services across regions with minute-level signals and burst events such as launches or campaigns.
• Partition forecasts by service, region, and horizon; isolate noisy services and coordinated dependency groups.
• Retain metrics, feature cutoffs, forecasts, actions, policy versions, outcomes, and cost evidence for replay.

• Peak scale: 10k services across 20 regions — Forecast and actuator volume drive dependency-aware queues and provider quota isolation.
• Latency target: forecast error < 15%; action lag < 2m — User-facing budget for the primary request or read path.
• Durable boundary: Committed before async — Observed capacity, versioned policies, forecasts, and actuator outcomes are authoritative; recommendations are derived.
• Async boundary: At-least-once workers — Keep Use time-series forecasting (Prophet, DeepAR) for demand prediction, Implement multi-step forecasting for different horizons, Use calendar features for event-driven scaling off the synchronous path.

Key entities

• DatasetVersiondatasetId, version, schemaHash, qualityStatus, lineage, createdAt

  Immutable predictive auto scaling system input version used for reproducible training, evaluation, or replay.

• FeatureSnapshotentityId, featureSetVersion, eventTime, values, sourceWatermarks

  Point-in-time predictive auto scaling system features with source watermarks so online and offline values can be compared.

• TrainingRunrunId, datasetVersion, codeVersion, metrics, artifactUri, status

  Audited predictive auto scaling system run that records data, code, dependency, and evaluation lineage.

• ModelVersionmodelId, version, stage, schema, qualityGates, endpoint

  A promotable predictive auto scaling system model version with rollout state, contract, and rollback metadata.

Data flow

1. 1. Register and validate training dataThe predictive auto scaling system gateway records an immutable dataset version, schema, lineage, quality status, and privacy disposition.
2. 2. Build point-in-time featuresFeature workers join predictive auto scaling system inputs using event-time watermarks, prevent leakage, and publish the same feature contract for training and serving.
3. 3. Train and evaluate asynchronouslyThe orchestrator schedules predictive auto scaling system runs with checkpointed artifacts, reproducible environments, and metrics tied to the exact input versions.
4. 4. Gate and serve a model versionA registry compares predictive auto scaling system quality, bias, safety, and compatibility gates before canary or production rollout with an immediate rollback pointer.
5. 5. Monitor drift and learn from feedbackOnline inference records latency, errors, drift, and delayed labels so predictive auto scaling system retraining is evidence-driven rather than triggered by guesswork.

Deep dives and trade-offs

• Reproducibility and leakage preventionPin predictive auto scaling system data, feature, code, dependency, and model versions for every run. Use point-in-time joins and quarantine failed quality or privacy checks before training. Keep raw inputs and artifacts immutable so a result can be replayed after a dependency changes.
• Safe promotion and serving contractsSeparate predictive auto scaling system model registration from deployment and require signed artifacts plus schema compatibility. Use shadow traffic, canaries, rollback pointers, and per-version latency/error budgets. Return model version and feature freshness so clients can explain or reproduce a prediction.
• Drift, feedback, and costMeasure feature drift, prediction drift, label delay, and segment-level quality for predictive auto scaling system rather than only aggregate accuracy. Sample expensive inference and cap retraining concurrency with an explicit GPU or compute budget. Keep human corrections and delayed labels linked to the original prediction and model version.
• Batch versus online featuresPrefer a shared feature contract with batch backfills and a low-latency online serving path for decisions that need freshness. Two independently defined transformations create training-serving skew and hard-to-debug regressions.
• Synchronous versus asynchronous inferenceKeep interactive predictive auto scaling system inference synchronous within a strict budget and queue large or expensive jobs. A request path that waits for model loading, enrichment, or retraining turns downstream slowness into an outage.
• Global model versus segment modelsStart with one versioned model and add segment-specific models only when quality or policy evidence justifies the operational cost. Many simultaneously active versions multiply monitoring, rollback, and data-lineage burden.