Design a Model Serving Infrastructure with Canary Deployments — System Design Interview Practice

Interview prompt

Design model serving infrastructure that exposes multiple ML frameworks through REST/gRPC, autoscaling GPU and CPU replicas, traffic splitting for canaries, and fast rollback based on health and quality signals.

• Make a signed deployment manifest authoritative for model digest, runtime, resource profile, routes, and rollout policy.
• Use weighted, sticky, or request-segmented traffic splitting with warmup, shadowing, and automatic rollback gates.
• Bound batching, queueing, feature lookups, and GPU memory; expose model and replica health separately from business quality.
• Explain multi-framework adapters, autoscaling, cold starts, overload, privacy, canary bias, and failure recovery.

Requirements and scale assumptions

• Register model deployments, provision replicas, serve REST/gRPC predictions, split traffic, and promote or roll back versions.
• Support batching, streaming responses, autoscaling, explainability, request limits, model-specific preprocessing, and health checks.
• Expose latency, errors, queue time, throughput, resource use, canary quality, drift, and rollout decision history.
• Target p99 inference latency below 100 ms for warm replicas and keep canary overhead within the request budget.
• Serve 100K requests per second across model versions with isolated queues and resource pools.
• Make deployment, route changes, prediction logging, and rollback idempotent; never split traffic to an unready model.
• Route to the last healthy version or return a bounded overload response when a pool or feature dependency fails.
• Operate 1,000 endpoints, 10,000 replicas, and 100K requests per second across CPU and GPU model pools.
• Partition routing and telemetry by endpoint, model version, tenant, and region; isolate hot endpoints.
• Retain signed deployment manifests, sampled predictions, rollout decisions, and health history while bounding logs.

• Peak scale: Support A/B testing between model versions — Capacity assumption that drives partitioning and backpressure.
• Latency target: P99 inference latency under 100ms — User-facing budget for the primary request or read path.
• Durable boundary: Committed before async — The source of truth is Serve models as REST/gRPC endpoints; Support canary deployments with gradual rollout.
• Async boundary: At-least-once workers — Keep Use KServe, Seldon, or Triton Inference Server, Implement Istio or Envoy for traffic management, Use Kubernetes HPA with custom metrics off the synchronous path.

Key entities

• DatasetVersiondatasetId, version, schemaHash, qualityStatus, lineage, createdAt

  Immutable model serving infrastructure with canary deployments input version used for reproducible training, evaluation, or replay.

• FeatureSnapshotentityId, featureSetVersion, eventTime, values, sourceWatermarks

  Point-in-time model serving infrastructure with canary deployments features with source watermarks so online and offline values can be compared.

• TrainingRunrunId, datasetVersion, codeVersion, metrics, artifactUri, status

  Audited model serving infrastructure with canary deployments run that records data, code, dependency, and evaluation lineage.

• ModelVersionmodelId, version, stage, schema, qualityGates, endpoint

  A promotable model serving infrastructure with canary deployments model version with rollout state, contract, and rollback metadata.

Data flow

1. 1. Register and validate training dataThe model serving infrastructure with canary deployments gateway records an immutable dataset version, schema, lineage, quality status, and privacy disposition.
2. 2. Build point-in-time featuresFeature workers join model serving infrastructure with canary deployments inputs using event-time watermarks, prevent leakage, and publish the same feature contract for training and serving.
3. 3. Train and evaluate asynchronouslyThe orchestrator schedules model serving infrastructure with canary deployments runs with checkpointed artifacts, reproducible environments, and metrics tied to the exact input versions.
4. 4. Gate and serve a model versionA registry compares model serving infrastructure with canary deployments quality, bias, safety, and compatibility gates before canary or production rollout with an immediate rollback pointer.
5. 5. Monitor drift and learn from feedbackOnline inference records latency, errors, drift, and delayed labels so model serving infrastructure with canary deployments retraining is evidence-driven rather than triggered by guesswork.

Deep dives and trade-offs

• Reproducibility and leakage preventionPin model serving infrastructure with canary deployments data, feature, code, dependency, and model versions for every run. Use point-in-time joins and quarantine failed quality or privacy checks before training. Keep raw inputs and artifacts immutable so a result can be replayed after a dependency changes.
• Safe promotion and serving contractsSeparate model serving infrastructure with canary deployments model registration from deployment and require signed artifacts plus schema compatibility. Use shadow traffic, canaries, rollback pointers, and per-version latency/error budgets. Return model version and feature freshness so clients can explain or reproduce a prediction.
• Drift, feedback, and costMeasure feature drift, prediction drift, label delay, and segment-level quality for model serving infrastructure with canary deployments rather than only aggregate accuracy. Sample expensive inference and cap retraining concurrency with an explicit GPU or compute budget. Keep human corrections and delayed labels linked to the original prediction and model version.
• Batch versus online featuresPrefer a shared feature contract with batch backfills and a low-latency online serving path for decisions that need freshness. Two independently defined transformations create training-serving skew and hard-to-debug regressions.
• Synchronous versus asynchronous inferenceKeep interactive model serving infrastructure with canary deployments inference synchronous within a strict budget and queue large or expensive jobs. A request path that waits for model loading, enrichment, or retraining turns downstream slowness into an outage.
• Global model versus segment modelsStart with one versioned model and add segment-specific models only when quality or policy evidence justifies the operational cost. Many simultaneously active versions multiply monitoring, rollback, and data-lineage burden.