Design a Machine Learning Platform — System Design Interview Practice

Interview prompt

Design an ML platform that manages datasets, features, experiments, training, deployment, explainability, monitoring, and governance from prototype through production.

• Version datasets, code, features, artifacts, evaluation results, approvals, and deployments as one reproducible lineage.
• Separate training and batch experimentation from low-latency online inference and enforce resource and tenant quotas.
• Make promotion depend on quality, fairness, safety, explainability, and reproducible signatures.
• Explain feature skew, drift, rollback, PII, model access, cost controls, and last-known-good fallback.

Requirements and scale assumptions

• Register data and features, launch experiments and training runs, evaluate models, and publish approved versions.
• Deploy models as batch or online endpoints, return explanations, and collect production feedback and drift signals.
• Support pipelines, approvals, lineage, model cards, access policy, rollback, retention, and reproducible exports.
• Target p95 online inference latency below 100 ms while long training and evaluation jobs remain asynchronous.
• Scale training, feature computation, and telemetry independently from online prediction traffic.
• Never deploy an unapproved artifact; make run creation, artifact publication, and metric ingestion idempotent.
• Keep the last healthy model serving when registry, feature, or new-model dependencies fail.
• Support 100K online predictions per second, thousands of experiments, and terabyte-scale training datasets.
• Partition runs and telemetry by tenant, project, model version, and time; isolate large backfills.
• Retain encrypted lineage, artifacts, explanations, approvals, and sampled inference events under policy.

• Peak scale: Explain model predictions — Capacity assumption that drives partitioning and backpressure.
• Latency target: Inference latency under 100ms — User-facing budget for the primary request or read path.
• Durable boundary: Committed before async — The source of truth is Train custom ML models; Use AutoML for quick prototypes.
• Async boundary: At-least-once workers — Keep Vertex AI for unified ML platform, AutoML for automated training, Cloud Functions for preprocessing off the synchronous path.

Key entities

• DatasetVersiondatasetId, version, schemaHash, qualityStatus, lineage, createdAt

  Immutable machine learning platform input version used for reproducible training, evaluation, or replay.

• FeatureSnapshotentityId, featureSetVersion, eventTime, values, sourceWatermarks

  Point-in-time machine learning platform features with source watermarks so online and offline values can be compared.

• TrainingRunrunId, datasetVersion, codeVersion, metrics, artifactUri, status

  Audited machine learning platform run that records data, code, dependency, and evaluation lineage.

• ModelVersionmodelId, version, stage, schema, qualityGates, endpoint

  A promotable machine learning platform model version with rollout state, contract, and rollback metadata.

Data flow

1. 1. Register and validate training dataThe machine learning platform gateway records an immutable dataset version, schema, lineage, quality status, and privacy disposition.
2. 2. Build point-in-time featuresFeature workers join machine learning platform inputs using event-time watermarks, prevent leakage, and publish the same feature contract for training and serving.
3. 3. Train and evaluate asynchronouslyThe orchestrator schedules machine learning platform runs with checkpointed artifacts, reproducible environments, and metrics tied to the exact input versions.
4. 4. Gate and serve a model versionA registry compares machine learning platform quality, bias, safety, and compatibility gates before canary or production rollout with an immediate rollback pointer.
5. 5. Monitor drift and learn from feedbackOnline inference records latency, errors, drift, and delayed labels so machine learning platform retraining is evidence-driven rather than triggered by guesswork.

Deep dives and trade-offs

• Reproducibility and leakage preventionPin machine learning platform data, feature, code, dependency, and model versions for every run. Use point-in-time joins and quarantine failed quality or privacy checks before training. Keep raw inputs and artifacts immutable so a result can be replayed after a dependency changes.
• Safe promotion and serving contractsSeparate machine learning platform model registration from deployment and require signed artifacts plus schema compatibility. Use shadow traffic, canaries, rollback pointers, and per-version latency/error budgets. Return model version and feature freshness so clients can explain or reproduce a prediction.
• Drift, feedback, and costMeasure feature drift, prediction drift, label delay, and segment-level quality for machine learning platform rather than only aggregate accuracy. Sample expensive inference and cap retraining concurrency with an explicit GPU or compute budget. Keep human corrections and delayed labels linked to the original prediction and model version.
• Batch versus online featuresPrefer a shared feature contract with batch backfills and a low-latency online serving path for decisions that need freshness. Two independently defined transformations create training-serving skew and hard-to-debug regressions.
• Synchronous versus asynchronous inferenceKeep interactive machine learning platform inference synchronous within a strict budget and queue large or expensive jobs. A request path that waits for model loading, enrichment, or retraining turns downstream slowness into an outage.
• Global model versus segment modelsStart with one versioned model and add segment-specific models only when quality or policy evidence justifies the operational cost. Many simultaneously active versions multiply monitoring, rollback, and data-lineage burden.