Build a Machine Learning Model Deployment Pipeline — System Design Interview Practice

Interview prompt

Design an end-to-end ML platform that trains models, validates artifacts, deploys them safely, supports A/B experiments, and monitors quality and latency in production.

• Version datasets, code, features, model artifacts, evaluation results, and deployment policy as one reproducible lineage.
• Separate expensive training from a low-latency inference fleet and use canaries or A/B routing for controlled rollout.
• Gate promotion on offline metrics, fairness and safety checks, resource limits, and reproducible signatures.
• Detect drift, rollback quickly, protect model and feature data, and make every run resumable and auditable.

Requirements and scale assumptions

• Register datasets and experiments, launch training, evaluate candidate models, and store immutable artifacts.
• Deploy a selected model to shadow, canary, A/B, and full-production stages with traffic and rollback controls.
• Collect inference metrics, feedback, drift signals, lineage, cost, and approval evidence for each deployment.
• Target p95 inference latency below 100 ms with warm replicas and bounded feature and model lookups.
• Scale training jobs and experiment telemetry independently from online inference traffic.
• Never deploy an unverified artifact; make training, evaluation, rollout, and metric ingestion idempotent.
• Keep the last healthy model available during registry, feature-store, or new-model failure.
• Run hundreds of experiments weekly, train on terabyte-scale datasets, and serve 100K inference requests per second.
• Partition experiment metadata by project and telemetry by model version and time; isolate high-volume producers.
• Retain immutable artifacts and sampled inference events while keeping online features and model replicas bounded.

• Inference traffic: 100K req/s — Online traffic drives replica autoscaling, batching limits, and model-cache capacity.
• Inference latency: p95 <=100ms — Online serving budget independent of long-running training and evaluation jobs.
• Durable boundary: Committed before async — The source of truth is an immutable model artifact plus its dataset, code, feature, and evaluation lineage.
• Async boundary: At-least-once workers — Keep training, evaluation, drift analysis, and registry updates off the inference path.

Key entities

• DatasetVersiondatasetId, version, schemaHash, qualityStatus, lineage, createdAt

  Immutable machine learning model deployment pipeline input version used for reproducible training, evaluation, or replay.

• FeatureSnapshotentityId, featureSetVersion, eventTime, values, sourceWatermarks

  Point-in-time machine learning model deployment pipeline features with source watermarks so online and offline values can be compared.

• TrainingRunrunId, datasetVersion, codeVersion, metrics, artifactUri, status

  Audited machine learning model deployment pipeline run that records data, code, dependency, and evaluation lineage.

• ModelVersionmodelId, version, stage, schema, qualityGates, endpoint

  A promotable machine learning model deployment pipeline model version with rollout state, contract, and rollback metadata.

Data flow

1. 1. Register and validate training dataThe machine learning model deployment pipeline gateway records an immutable dataset version, schema, lineage, quality status, and privacy disposition.
2. 2. Build point-in-time featuresFeature workers join machine learning model deployment pipeline inputs using event-time watermarks, prevent leakage, and publish the same feature contract for training and serving.
3. 3. Train and evaluate asynchronouslyThe orchestrator schedules machine learning model deployment pipeline runs with checkpointed artifacts, reproducible environments, and metrics tied to the exact input versions.
4. 4. Gate and serve a model versionA registry compares machine learning model deployment pipeline quality, bias, safety, and compatibility gates before canary or production rollout with an immediate rollback pointer.
5. 5. Monitor drift and learn from feedbackOnline inference records latency, errors, drift, and delayed labels so machine learning model deployment pipeline retraining is evidence-driven rather than triggered by guesswork.

Deep dives and trade-offs

• Reproducibility and leakage preventionPin machine learning model deployment pipeline data, feature, code, dependency, and model versions for every run. Use point-in-time joins and quarantine failed quality or privacy checks before training. Keep raw inputs and artifacts immutable so a result can be replayed after a dependency changes.
• Safe promotion and serving contractsSeparate machine learning model deployment pipeline model registration from deployment and require signed artifacts plus schema compatibility. Use shadow traffic, canaries, rollback pointers, and per-version latency/error budgets. Return model version and feature freshness so clients can explain or reproduce a prediction.
• Drift, feedback, and costMeasure feature drift, prediction drift, label delay, and segment-level quality for machine learning model deployment pipeline rather than only aggregate accuracy. Sample expensive inference and cap retraining concurrency with an explicit GPU or compute budget. Keep human corrections and delayed labels linked to the original prediction and model version.
• Batch versus online featuresPrefer a shared feature contract with batch backfills and a low-latency online serving path for decisions that need freshness. Two independently defined transformations create training-serving skew and hard-to-debug regressions.
• Synchronous versus asynchronous inferenceKeep interactive machine learning model deployment pipeline inference synchronous within a strict budget and queue large or expensive jobs. A request path that waits for model loading, enrichment, or retraining turns downstream slowness into an outage.
• Global model versus segment modelsStart with one versioned model and add segment-specific models only when quality or policy evidence justifies the operational cost. Many simultaneously active versions multiply monitoring, rollback, and data-lineage burden.