Design a Feature Store for Machine Learning — System Design Interview Practice

Interview prompt

Design a feature store that computes, governs, and serves features consistently for offline training and low-latency online inference, with freshness, lineage, and point-in-time correctness.

• Define feature ownership, entity keys, transformation code, event-time semantics, freshness SLA, schema, and deprecation policy.
• Use one transformation definition for offline and online paths, prevent training-serving skew, and enforce point-in-time joins.
• Separate the low-latency online store from the durable offline store and feature registry, with replayable materialization.
• Explain late events, backfills, missing features, access control, lineage, PII, hot keys, and safe model fallback.

Requirements and scale assumptions

• Register and discover features, define transformations, materialize historical values, and backfill online entity values.
• Read a batch of online features for inference and retrieve point-in-time correct feature sets for training.
• Expose freshness, lineage, schema, ownership, quality, access policy, usage, deprecation, and materialization status.
• Target p95 online feature reads below 10 ms for a bounded batch and expose freshness rather than silently serving unknown values.
• Serve 100,000 requests per second and materialize millions of entity-feature values through partitioned workers.
• Make event processing and materialization idempotent, version schemas, and preserve offline snapshots for reproducibility.
• Return a last-known value or explicit missing-feature signal when online storage or an upstream stream is degraded.
• Support 10,000 registered features, 50 million entities, and hundreds of model-serving consumers.
• Partition online values by entity key and feature namespace; isolate hot entities and high-volume streams.
• Retain versioned offline values, transformation code, lineage, quality results, and materialization checkpoints.

• Peak scale: Support feature sharing across teams — Capacity assumption that drives partitioning and backpressure.
• Latency target: Online feature serving latency under 10ms — User-facing budget for the primary request or read path.
• Durable boundary: Committed before async — The source of truth is Serve features for batch training and online inference; Ensure offline-online feature consistency.
• Async boundary: At-least-once workers — Keep Use Feast, Tecton, or custom implementation, Implement dual storage (Redis for online, S3/BigQuery for offline), Use streaming frameworks for real-time feature computation off the synchronous path.

Key entities

• DatasetVersiondatasetId, version, schemaHash, qualityStatus, lineage, createdAt

  Immutable feature store input version used for reproducible training, evaluation, or replay.

• FeatureSnapshotentityId, featureSetVersion, eventTime, values, sourceWatermarks

  Point-in-time feature store features with source watermarks so online and offline values can be compared.

• TrainingRunrunId, datasetVersion, codeVersion, metrics, artifactUri, status

  Audited feature store run that records data, code, dependency, and evaluation lineage.

• ModelVersionmodelId, version, stage, schema, qualityGates, endpoint

  A promotable feature store model version with rollout state, contract, and rollback metadata.

Data flow

1. 1. Register and validate training dataThe feature store gateway records an immutable dataset version, schema, lineage, quality status, and privacy disposition.
2. 2. Build point-in-time featuresFeature workers join feature store inputs using event-time watermarks, prevent leakage, and publish the same feature contract for training and serving.
3. 3. Train and evaluate asynchronouslyThe orchestrator schedules feature store runs with checkpointed artifacts, reproducible environments, and metrics tied to the exact input versions.
4. 4. Gate and serve a model versionA registry compares feature store quality, bias, safety, and compatibility gates before canary or production rollout with an immediate rollback pointer.
5. 5. Monitor drift and learn from feedbackOnline inference records latency, errors, drift, and delayed labels so feature store retraining is evidence-driven rather than triggered by guesswork.

Deep dives and trade-offs

• Reproducibility and leakage preventionPin feature store data, feature, code, dependency, and model versions for every run. Use point-in-time joins and quarantine failed quality or privacy checks before training. Keep raw inputs and artifacts immutable so a result can be replayed after a dependency changes.
• Safe promotion and serving contractsSeparate feature store model registration from deployment and require signed artifacts plus schema compatibility. Use shadow traffic, canaries, rollback pointers, and per-version latency/error budgets. Return model version and feature freshness so clients can explain or reproduce a prediction.
• Drift, feedback, and costMeasure feature drift, prediction drift, label delay, and segment-level quality for feature store rather than only aggregate accuracy. Sample expensive inference and cap retraining concurrency with an explicit GPU or compute budget. Keep human corrections and delayed labels linked to the original prediction and model version.
• Batch versus online featuresPrefer a shared feature contract with batch backfills and a low-latency online serving path for decisions that need freshness. Two independently defined transformations create training-serving skew and hard-to-debug regressions.
• Synchronous versus asynchronous inferenceKeep interactive feature store inference synchronous within a strict budget and queue large or expensive jobs. A request path that waits for model loading, enrichment, or retraining turns downstream slowness into an outage.
• Global model versus segment modelsStart with one versioned model and add segment-specific models only when quality or policy evidence justifies the operational cost. Many simultaneously active versions multiply monitoring, rollback, and data-lineage burden.