Design a Model Registry and Versioning System — System Design Interview Practice

Interview prompt

Design a model registry that immutably stores artifacts and metadata, tracks lineage and model cards, manages lifecycle stages, and gates production promotion with approvals and policy checks.

• Make an immutable artifact digest plus dataset, code, feature, evaluation, owner, and approval metadata the version authority.
• Separate metadata queries from large artifact storage and make promotion a conditional lifecycle transition.
• Integrate CI/CD and serving systems through signed manifests, policy gates, webhooks, and idempotent callbacks.
• Explain artifact retention, lineage, access controls, rollback, deletion, registry outage, and audit evidence.

Requirements and scale assumptions

• Register artifacts, metadata, model cards, evaluation results, dependencies, and immutable versions.
• Move versions through draft, validated, staging, canary, production, archived, and rejected lifecycle stages.
• Support approvals, policy checks, deployment webhooks, artifact downloads, rollback, deprecation, and audit export.
• Support 100,000 model versions with metadata lookups below 200 ms p95 and immutable artifact downloads through signed URLs.
• Partition metadata by tenant, project, model name, and version while isolating high-volume CI/CD integrations.
• Make registration, approval, promotion, webhook delivery, and artifact garbage collection idempotent.
• Serve the last approved production manifest when new registry writes or evaluation systems are unavailable.
• Track 100,000 versions, 10,000 active models, and CI/CD events from hundreds of repositories.
• Partition by tenant, project, model, and lifecycle stage; isolate large artifact transfers from metadata reads.
• Retain signed manifests, lineage, approvals, model cards, audit logs, and artifact retention decisions.

• Peak scale: Integrate with CI/CD for automated deployment — Capacity assumption that drives partitioning and backpressure.
• Latency target: Support 10k+ model versions — User-facing budget for the primary request or read path.
• Durable boundary: Committed before async — The source of truth is Register and version ML models; Store model artifacts and metadata.
• Async boundary: At-least-once workers — Keep Use MLflow or Weights & Biases as a base, Store artifacts in object storage with metadata in DB, Implement model cards for documentation off the synchronous path.

Key entities

• DatasetVersiondatasetId, version, schemaHash, qualityStatus, lineage, createdAt

  Immutable model registry and versioning system input version used for reproducible training, evaluation, or replay.

• FeatureSnapshotentityId, featureSetVersion, eventTime, values, sourceWatermarks

  Point-in-time model registry and versioning system features with source watermarks so online and offline values can be compared.

• TrainingRunrunId, datasetVersion, codeVersion, metrics, artifactUri, status

  Audited model registry and versioning system run that records data, code, dependency, and evaluation lineage.

• ModelVersionmodelId, version, stage, schema, qualityGates, endpoint

  A promotable model registry and versioning system model version with rollout state, contract, and rollback metadata.

Data flow

1. 1. Register and validate training dataThe model registry and versioning system gateway records an immutable dataset version, schema, lineage, quality status, and privacy disposition.
2. 2. Build point-in-time featuresFeature workers join model registry and versioning system inputs using event-time watermarks, prevent leakage, and publish the same feature contract for training and serving.
3. 3. Train and evaluate asynchronouslyThe orchestrator schedules model registry and versioning system runs with checkpointed artifacts, reproducible environments, and metrics tied to the exact input versions.
4. 4. Gate and serve a model versionA registry compares model registry and versioning system quality, bias, safety, and compatibility gates before canary or production rollout with an immediate rollback pointer.
5. 5. Monitor drift and learn from feedbackOnline inference records latency, errors, drift, and delayed labels so model registry and versioning system retraining is evidence-driven rather than triggered by guesswork.

Deep dives and trade-offs

• Reproducibility and leakage preventionPin model registry and versioning system data, feature, code, dependency, and model versions for every run. Use point-in-time joins and quarantine failed quality or privacy checks before training. Keep raw inputs and artifacts immutable so a result can be replayed after a dependency changes.
• Safe promotion and serving contractsSeparate model registry and versioning system model registration from deployment and require signed artifacts plus schema compatibility. Use shadow traffic, canaries, rollback pointers, and per-version latency/error budgets. Return model version and feature freshness so clients can explain or reproduce a prediction.
• Drift, feedback, and costMeasure feature drift, prediction drift, label delay, and segment-level quality for model registry and versioning system rather than only aggregate accuracy. Sample expensive inference and cap retraining concurrency with an explicit GPU or compute budget. Keep human corrections and delayed labels linked to the original prediction and model version.
• Batch versus online featuresPrefer a shared feature contract with batch backfills and a low-latency online serving path for decisions that need freshness. Two independently defined transformations create training-serving skew and hard-to-debug regressions.
• Synchronous versus asynchronous inferenceKeep interactive model registry and versioning system inference synchronous within a strict budget and queue large or expensive jobs. A request path that waits for model loading, enrichment, or retraining turns downstream slowness into an outage.
• Global model versus segment modelsStart with one versioned model and add segment-specific models only when quality or policy evidence justifies the operational cost. Many simultaneously active versions multiply monitoring, rollback, and data-lineage burden.