Design a Credit Scoring ML Pipeline — System Design Interview Practice

Interview prompt

Design a regulated credit-scoring ML platform that trains on historical repayment data, handles class imbalance, serves explainable decisions, monitors drift and fairness, and supports model governance.

• Version source snapshots, feature definitions, labels, training code, model artifacts, approvals, and decision explanations.
• Prevent leakage and sampling bias, evaluate imbalanced outcomes and subgroup fairness, and preserve reproducible training lineage.
• Gate deployment on offline metrics, explainability, stability, policy approval, and challenger comparison.
• Explain PII controls, adverse-action reasons, drift, appeals, rollback, retention, and independent audit access.

Requirements and scale assumptions

• Create governed datasets, train and evaluate candidates, compare subgroup metrics, and register approved model versions.
• Score an application with a bounded online feature lookup and return a decision, reason codes, model version, and trace.
• Support human review, appeals, adverse-action notices, monitoring, rollback, deletion requests, and compliance reports.
• Target AUC-ROC above 0.85 only as one measure; also enforce decision latency below 200 ms and fairness thresholds.
• Scale training and evaluation separately from online scoring, with tenant and model-version isolation.
• Never serve an unapproved or unverifiable model; make feature reads, decisions, and report generation traceable.
• Fall back to the last approved model or manual review when features, registry, or a new model is unavailable.
• Train on millions of applications, serve 50,000 scoring requests per second, and retain governed evidence for seven years.
• Partition training data and telemetry by model version, time, and tenant; isolate large backfills.
• Retain encrypted feature snapshots, labels, explanations, approvals, and decisions while minimizing online state.

• Peak scale: Produce regulatory compliance documentation — Capacity assumption that drives partitioning and backpressure.
• Latency target: AUC-ROC above 0.85 — User-facing budget for the primary request or read path.
• Durable boundary: Committed before async — The source of truth is Train credit scoring models on historical data; Handle imbalanced classes (default vs non-default).
• Async boundary: At-least-once workers — Keep Use gradient boosting (XGBoost, LightGBM) as baseline, Implement SHAP values for model explainability, Use SMOTE or class weights for imbalanced data off the synchronous path.

Key entities

• DatasetVersiondatasetId, version, schemaHash, qualityStatus, lineage, createdAt

  Immutable credit scoring ml pipeline input version used for reproducible training, evaluation, or replay.

• FeatureSnapshotentityId, featureSetVersion, eventTime, values, sourceWatermarks

  Point-in-time credit scoring ml pipeline features with source watermarks so online and offline values can be compared.

• TrainingRunrunId, datasetVersion, codeVersion, metrics, artifactUri, status

  Audited credit scoring ml pipeline run that records data, code, dependency, and evaluation lineage.

• ModelVersionmodelId, version, stage, schema, qualityGates, endpoint

  A promotable credit scoring ml pipeline model version with rollout state, contract, and rollback metadata.

Data flow

1. 1. Register and validate training dataThe credit scoring ml pipeline gateway records an immutable dataset version, schema, lineage, quality status, and privacy disposition.
2. 2. Build point-in-time featuresFeature workers join credit scoring ml pipeline inputs using event-time watermarks, prevent leakage, and publish the same feature contract for training and serving.
3. 3. Train and evaluate asynchronouslyThe orchestrator schedules credit scoring ml pipeline runs with checkpointed artifacts, reproducible environments, and metrics tied to the exact input versions.
4. 4. Gate and serve a model versionA registry compares credit scoring ml pipeline quality, bias, safety, and compatibility gates before canary or production rollout with an immediate rollback pointer.
5. 5. Monitor drift and learn from feedbackOnline inference records latency, errors, drift, and delayed labels so credit scoring ml pipeline retraining is evidence-driven rather than triggered by guesswork.

Deep dives and trade-offs

• Reproducibility and leakage preventionPin credit scoring ml pipeline data, feature, code, dependency, and model versions for every run. Use point-in-time joins and quarantine failed quality or privacy checks before training. Keep raw inputs and artifacts immutable so a result can be replayed after a dependency changes.
• Safe promotion and serving contractsSeparate credit scoring ml pipeline model registration from deployment and require signed artifacts plus schema compatibility. Use shadow traffic, canaries, rollback pointers, and per-version latency/error budgets. Return model version and feature freshness so clients can explain or reproduce a prediction.
• Drift, feedback, and costMeasure feature drift, prediction drift, label delay, and segment-level quality for credit scoring ml pipeline rather than only aggregate accuracy. Sample expensive inference and cap retraining concurrency with an explicit GPU or compute budget. Keep human corrections and delayed labels linked to the original prediction and model version.
• Batch versus online featuresPrefer a shared feature contract with batch backfills and a low-latency online serving path for decisions that need freshness. Two independently defined transformations create training-serving skew and hard-to-debug regressions.
• Synchronous versus asynchronous inferenceKeep interactive credit scoring ml pipeline inference synchronous within a strict budget and queue large or expensive jobs. A request path that waits for model loading, enrichment, or retraining turns downstream slowness into an outage.
• Global model versus segment modelsStart with one versioned model and add segment-specific models only when quality or policy evidence justifies the operational cost. Many simultaneously active versions multiply monitoring, rollback, and data-lineage burden.