Design an AI-Driven Change Risk Assessment System — System Design Interview Practice

Interview prompt

Design a change-risk service that analyzes proposed code, configuration, and infrastructure changes against service topology and incident history, then recommends guarded rollout plans and windows.

• Define change identity, diff/config features, service ownership/topology, historical incident labels, risk explanations, and model versions.
• Score blast radius and collision risk without blocking all delivery; distinguish evidence-backed risk from uncertainty and recommend canaries/approvals.
• Separate CI/CD admission from asynchronous feature building and incident correlation; make scores reproducible and idempotent.
• Explain feedback loops, concept drift, sensitive code, overrides, rollback, auditability, observability, and rule-based fallback.

Requirements and scale assumptions

• Ingest proposed changes and context, compute affected services/features, score risk, explain drivers, and recommend rollout gates or windows.
• Integrate with CI/CD and change management, record deployment outcomes/incidents, support approvals, overrides, canary progression, and rollback.
• Version features/models, protect source data, replay historical changes, delete sensitive evidence, and survive unavailable topology or model services.
• Return a preliminary risk score within 30 seconds and never bypass mandatory policy gates when the ML service is unavailable.
• Score 100k changes per day across thousands of services without a single hot key or unbounded synchronous work.
• Do not lose committed state; make retries and duplicate events safe.
• Degrade safely when downstream workers, caches, or external dependencies fail.
• 100k changes/day across 10k services and 1M historical incidents
• Partition by the primary tenant, user, item, or geographic key and isolate hot partitions.
• Keep serving state bounded; retain raw events or durable records for replay and auditing.

• Peak scale: 100k changes/day; 10k services — Capacity assumption that drives partitioning and backpressure.
• Latency target: preliminary score < 30s; policy fail-closed — User-facing budget for the primary request or read path.
• Durable boundary: Committed before async — Versioned change records, deployment outcomes, and incidents are authoritative; scores are derived evidence.
• Async boundary: At-least-once workers — Keep Use gradient boosting models for risk classification, Extract features from code diffs, service dependencies, timing, Implement change collision detection algorithms off the synchronous path.

Key entities

• DatasetVersiondatasetId, version, schemaHash, qualityStatus, lineage, createdAt

  Immutable ai driven change risk assessment system input version used for reproducible training, evaluation, or replay.

• FeatureSnapshotentityId, featureSetVersion, eventTime, values, sourceWatermarks

  Point-in-time ai driven change risk assessment system features with source watermarks so online and offline values can be compared.

• TrainingRunrunId, datasetVersion, codeVersion, metrics, artifactUri, status

  Audited ai driven change risk assessment system run that records data, code, dependency, and evaluation lineage.

• ModelVersionmodelId, version, stage, schema, qualityGates, endpoint

  A promotable ai driven change risk assessment system model version with rollout state, contract, and rollback metadata.

Data flow

1. 1. Register and validate training dataThe ai driven change risk assessment system gateway records an immutable dataset version, schema, lineage, quality status, and privacy disposition.
2. 2. Build point-in-time featuresFeature workers join ai driven change risk assessment system inputs using event-time watermarks, prevent leakage, and publish the same feature contract for training and serving.
3. 3. Train and evaluate asynchronouslyThe orchestrator schedules ai driven change risk assessment system runs with checkpointed artifacts, reproducible environments, and metrics tied to the exact input versions.
4. 4. Gate and serve a model versionA registry compares ai driven change risk assessment system quality, bias, safety, and compatibility gates before canary or production rollout with an immediate rollback pointer.
5. 5. Monitor drift and learn from feedbackOnline inference records latency, errors, drift, and delayed labels so ai driven change risk assessment system retraining is evidence-driven rather than triggered by guesswork.

Deep dives and trade-offs

• Reproducibility and leakage preventionPin ai driven change risk assessment system data, feature, code, dependency, and model versions for every run. Use point-in-time joins and quarantine failed quality or privacy checks before training. Keep raw inputs and artifacts immutable so a result can be replayed after a dependency changes.
• Safe promotion and serving contractsSeparate ai driven change risk assessment system model registration from deployment and require signed artifacts plus schema compatibility. Use shadow traffic, canaries, rollback pointers, and per-version latency/error budgets. Return model version and feature freshness so clients can explain or reproduce a prediction.
• Drift, feedback, and costMeasure feature drift, prediction drift, label delay, and segment-level quality for ai driven change risk assessment system rather than only aggregate accuracy. Sample expensive inference and cap retraining concurrency with an explicit GPU or compute budget. Keep human corrections and delayed labels linked to the original prediction and model version.
• Batch versus online featuresPrefer a shared feature contract with batch backfills and a low-latency online serving path for decisions that need freshness. Two independently defined transformations create training-serving skew and hard-to-debug regressions.
• Synchronous versus asynchronous inferenceKeep interactive ai driven change risk assessment system inference synchronous within a strict budget and queue large or expensive jobs. A request path that waits for model loading, enrichment, or retraining turns downstream slowness into an outage.
• Global model versus segment modelsStart with one versioned model and add segment-specific models only when quality or policy evidence justifies the operational cost. Many simultaneously active versions multiply monitoring, rollback, and data-lineage burden.