Design a Data Pipeline for ML with Data Quality Gates — System Design Interview Practice

Interview prompt

Design a reproducible ML data pipeline that ingests heterogeneous sources, enforces schema and quality gates, versions point-in-time datasets, and blocks unsafe training data from publication.

• Define source contracts, schema versions, raw/validated/feature zones, data-cutoff time, lineage, quality rules, and quarantine semantics.
• Make validation gates measurable for completeness, freshness, ranges, distributions, labels, duplicates, and leakage before training.
• Separate ingestion, transformation, validation, feature materialization, and dataset publication; make every run replayable and idempotent.
• Explain late data, backfills, schema evolution, PII, cost, approvals, observability, and safe handling of failed or stale runs.

Requirements and scale assumptions

• Ingest database changes, APIs, and files; validate contracts; transform and join data; compute features; and publish versioned datasets.
• Expose run status, lineage, rule results, freshness, schema diffs, quarantined records, dataset manifests, and approval history.
• Support replay/backfill, correction, deletion propagation, rollback to a prior dataset, retries, and training-trigger integration.
• Complete daily production runs within two hours and fail a training trigger before publishing data that violates critical gates.
• Process 100TB daily across thousands of features without a single hot key or unbounded synchronous work.
• Do not lose committed state; make retries and duplicate events safe.
• Degrade safely when downstream workers, caches, or external dependencies fail.
• 100TB/day, 10k features, and 1k datasets
• Partition by the primary tenant, user, item, or geographic key and isolate hot partitions.
• Keep serving state bounded; retain raw events or durable records for replay and auditing.

• Peak scale: 100TB/day; 1k datasets — Capacity assumption that drives partitioning and backpressure.
• Latency target: daily run < 2h; critical gates block publish — User-facing budget for the primary request or read path.
• Durable boundary: Committed before async — Immutable source snapshots and approved dataset manifests are authoritative; transformations and reports are derived.
• Async boundary: At-least-once workers — Keep Use Great Expectations or Pandera for data validation, Implement DVC or LakeFS for data versioning, Use Apache Spark or dbt for data transformations off the synchronous path.

Key entities

• DatasetVersiondatasetId, version, schemaHash, qualityStatus, lineage, createdAt

  Immutable data pipeline input version used for reproducible training, evaluation, or replay.

• FeatureSnapshotentityId, featureSetVersion, eventTime, values, sourceWatermarks

  Point-in-time data pipeline features with source watermarks so online and offline values can be compared.

• TrainingRunrunId, datasetVersion, codeVersion, metrics, artifactUri, status

  Audited data pipeline run that records data, code, dependency, and evaluation lineage.

• ModelVersionmodelId, version, stage, schema, qualityGates, endpoint

  A promotable data pipeline model version with rollout state, contract, and rollback metadata.

Data flow

1. 1. Register and validate training dataThe data pipeline gateway records an immutable dataset version, schema, lineage, quality status, and privacy disposition.
2. 2. Build point-in-time featuresFeature workers join data pipeline inputs using event-time watermarks, prevent leakage, and publish the same feature contract for training and serving.
3. 3. Train and evaluate asynchronouslyThe orchestrator schedules data pipeline runs with checkpointed artifacts, reproducible environments, and metrics tied to the exact input versions.
4. 4. Gate and serve a model versionA registry compares data pipeline quality, bias, safety, and compatibility gates before canary or production rollout with an immediate rollback pointer.
5. 5. Monitor drift and learn from feedbackOnline inference records latency, errors, drift, and delayed labels so data pipeline retraining is evidence-driven rather than triggered by guesswork.

Deep dives and trade-offs

• Reproducibility and leakage preventionPin data pipeline data, feature, code, dependency, and model versions for every run. Use point-in-time joins and quarantine failed quality or privacy checks before training. Keep raw inputs and artifacts immutable so a result can be replayed after a dependency changes.
• Safe promotion and serving contractsSeparate data pipeline model registration from deployment and require signed artifacts plus schema compatibility. Use shadow traffic, canaries, rollback pointers, and per-version latency/error budgets. Return model version and feature freshness so clients can explain or reproduce a prediction.
• Drift, feedback, and costMeasure feature drift, prediction drift, label delay, and segment-level quality for data pipeline rather than only aggregate accuracy. Sample expensive inference and cap retraining concurrency with an explicit GPU or compute budget. Keep human corrections and delayed labels linked to the original prediction and model version.
• Batch versus online featuresPrefer a shared feature contract with batch backfills and a low-latency online serving path for decisions that need freshness. Two independently defined transformations create training-serving skew and hard-to-debug regressions.
• Synchronous versus asynchronous inferenceKeep interactive data pipeline inference synchronous within a strict budget and queue large or expensive jobs. A request path that waits for model loading, enrichment, or retraining turns downstream slowness into an outage.
• Global model versus segment modelsStart with one versioned model and add segment-specific models only when quality or policy evidence justifies the operational cost. Many simultaneously active versions multiply monitoring, rollback, and data-lineage burden.