Design an ML Experiment Tracking System — System Design Interview Practice

Interview prompt

Design a collaborative ML experiment-tracking service that records parameters, metrics, code/data/environment lineage, artifacts, and evaluation results so runs can be compared and reproduced.

• Define projects, runs, parameters, time-series metrics, source/code/data/environment lineage, artifacts, tags, comparisons, and lifecycle states.
• Keep high-frequency logging append-only and resumable; separate metadata/query indexes from large artifact storage and enforce quotas/cardinality.
• Support framework SDKs, offline buffering, run finalization, immutable versions, reproducible environment capture, and artifact integrity checks.
• Explain tenant access, secret redaction, deletion/retention, concurrent logging, retries, outage recovery, observability, and local buffering.

Requirements and scale assumptions

• Create projects/runs, log parameters and metrics, upload artifacts, capture lineage, finalize runs, and compare/search runs.
• Support dashboards, charts, tags, framework SDKs, resumable uploads, artifact download, run notes, and model-registration handoff.
• Make logging idempotent, preserve append-only history, support retention/deletion, access controls, export, and recovery after agent disconnects.
• Acknowledge metric batches quickly and make run metadata searchable while large artifact uploads remain asynchronous.
• Scale to 100k active runs, 1M metric points per second, and large artifacts without a single hot key or unbounded synchronous work.
• Do not lose committed state; make retries and duplicate events safe.
• Degrade safely when downstream workers, caches, or external dependencies fail.
• 100k active runs, 1M metric points/second, and petabytes of artifacts
• Partition by the primary tenant, user, item, or geographic key and isolate hot partitions.
• Keep serving state bounded; retain raw events or durable records for replay and auditing.

• Peak scale: 100k runs; 1M metric points/s — Capacity assumption that drives partitioning and backpressure.
• Latency target: metric batch ack < 1s; artifacts async — User-facing budget for the primary request or read path.
• Durable boundary: Committed before async — Run metadata, append-only metric logs, lineage, and artifact manifests are authoritative; indexes are derived.
• Async boundary: At-least-once workers — Keep Use MLflow, Weights & Biases, or Neptune as reference, Implement append-only log storage for metrics, Use object storage for large artifacts off the synchronous path.

Key entities

• DatasetVersiondatasetId, version, schemaHash, qualityStatus, lineage, createdAt

  Immutable ml experiment tracking system input version used for reproducible training, evaluation, or replay.

• FeatureSnapshotentityId, featureSetVersion, eventTime, values, sourceWatermarks

  Point-in-time ml experiment tracking system features with source watermarks so online and offline values can be compared.

• TrainingRunrunId, datasetVersion, codeVersion, metrics, artifactUri, status

  Audited ml experiment tracking system run that records data, code, dependency, and evaluation lineage.

• ModelVersionmodelId, version, stage, schema, qualityGates, endpoint

  A promotable ml experiment tracking system model version with rollout state, contract, and rollback metadata.

Data flow

1. 1. Register and validate training dataThe ml experiment tracking system gateway records an immutable dataset version, schema, lineage, quality status, and privacy disposition.
2. 2. Build point-in-time featuresFeature workers join ml experiment tracking system inputs using event-time watermarks, prevent leakage, and publish the same feature contract for training and serving.
3. 3. Train and evaluate asynchronouslyThe orchestrator schedules ml experiment tracking system runs with checkpointed artifacts, reproducible environments, and metrics tied to the exact input versions.
4. 4. Gate and serve a model versionA registry compares ml experiment tracking system quality, bias, safety, and compatibility gates before canary or production rollout with an immediate rollback pointer.
5. 5. Monitor drift and learn from feedbackOnline inference records latency, errors, drift, and delayed labels so ml experiment tracking system retraining is evidence-driven rather than triggered by guesswork.

Deep dives and trade-offs

• Reproducibility and leakage preventionPin ml experiment tracking system data, feature, code, dependency, and model versions for every run. Use point-in-time joins and quarantine failed quality or privacy checks before training. Keep raw inputs and artifacts immutable so a result can be replayed after a dependency changes.
• Safe promotion and serving contractsSeparate ml experiment tracking system model registration from deployment and require signed artifacts plus schema compatibility. Use shadow traffic, canaries, rollback pointers, and per-version latency/error budgets. Return model version and feature freshness so clients can explain or reproduce a prediction.
• Drift, feedback, and costMeasure feature drift, prediction drift, label delay, and segment-level quality for ml experiment tracking system rather than only aggregate accuracy. Sample expensive inference and cap retraining concurrency with an explicit GPU or compute budget. Keep human corrections and delayed labels linked to the original prediction and model version.
• Batch versus online featuresPrefer a shared feature contract with batch backfills and a low-latency online serving path for decisions that need freshness. Two independently defined transformations create training-serving skew and hard-to-debug regressions.
• Synchronous versus asynchronous inferenceKeep interactive ml experiment tracking system inference synchronous within a strict budget and queue large or expensive jobs. A request path that waits for model loading, enrichment, or retraining turns downstream slowness into an outage.
• Global model versus segment modelsStart with one versioned model and add segment-specific models only when quality or policy evidence justifies the operational cost. Many simultaneously active versions multiply monitoring, rollback, and data-lineage burden.