Design an Autonomous Vehicle Perception System — System Design Interview Practice

Interview prompt

Design a safety-critical vehicle-perception stack that time-synchronizes camera, LiDAR, and radar data, detects and tracks 3D objects, estimates motion, and exposes confidence and sensor-health state to planning.

• Define sensor calibration/time sync, frame IDs, object tracks, classes, covariance/confidence, occlusion, free space, and safety contracts.
• Bound compute and memory for deterministic latency; detect sensor degradation, out-of-distribution scenes, stale frames, and conflicting modalities.
• Separate safety-critical online inference from dataset logging, labeling, training, replay, simulation, and model rollout validation.
• Explain redundancy, fail-safe behavior, uncertainty propagation, privacy, hardware failure, observability, and minimum-risk degraded perception.

Requirements and scale assumptions

• Capture synchronized sensor frames, calibrate/fuse them, detect 3D objects/free space, track motion, and publish timestamped scene hypotheses.
• Expose confidence, sensor health, track history, occlusions, latency, and uncertainty to planning/control with explicit safety limits.
• Support model/configuration versioning, replay/simulation, secure fleet telemetry, rollback, sensor dropouts, and incident evidence capture.
• Meet deterministic end-to-end perception latency under 100ms with timestamp integrity and bounded behavior under sensor degradation.
• Process 30 frames per second across a fleet while keeping training/replay workloads off the vehicle critical path.
• Do not lose committed state; make retries and duplicate events safe.
• Degrade safely when downstream workers, caches, or external dependencies fail.
• 30 FPS sensor fusion per vehicle across a 100k-vehicle fleet
• Partition by the primary tenant, user, item, or geographic key and isolate hot partitions.
• Keep serving state bounded; retain raw events or durable records for replay and auditing.

• Peak scale: 30 FPS; 100k vehicles — Capacity assumption that drives partitioning and backpressure.
• Latency target: perception p99 < 100ms; health explicit — User-facing budget for the primary request or read path.
• Durable boundary: Committed before async — Timestamped sensor frames and calibrated vehicle configuration are authoritative; tracks and detections are derived.
• Async boundary: At-least-once workers — Keep Use BEV (Bird's Eye View) representations for fusion, Implement PointPillars or CenterPoint for LiDAR detection, Use temporal fusion for multi-frame tracking off the synchronous path.

Key entities

• DatasetVersiondatasetId, version, schemaHash, qualityStatus, lineage, createdAt

  Immutable autonomous vehicle perception system input version used for reproducible training, evaluation, or replay.

• FeatureSnapshotentityId, featureSetVersion, eventTime, values, sourceWatermarks

  Point-in-time autonomous vehicle perception system features with source watermarks so online and offline values can be compared.

• TrainingRunrunId, datasetVersion, codeVersion, metrics, artifactUri, status

  Audited autonomous vehicle perception system run that records data, code, dependency, and evaluation lineage.

• ModelVersionmodelId, version, stage, schema, qualityGates, endpoint

  A promotable autonomous vehicle perception system model version with rollout state, contract, and rollback metadata.

Data flow

1. 1. Register and validate training dataThe autonomous vehicle perception system gateway records an immutable dataset version, schema, lineage, quality status, and privacy disposition.
2. 2. Build point-in-time featuresFeature workers join autonomous vehicle perception system inputs using event-time watermarks, prevent leakage, and publish the same feature contract for training and serving.
3. 3. Train and evaluate asynchronouslyThe orchestrator schedules autonomous vehicle perception system runs with checkpointed artifacts, reproducible environments, and metrics tied to the exact input versions.
4. 4. Gate and serve a model versionA registry compares autonomous vehicle perception system quality, bias, safety, and compatibility gates before canary or production rollout with an immediate rollback pointer.
5. 5. Monitor drift and learn from feedbackOnline inference records latency, errors, drift, and delayed labels so autonomous vehicle perception system retraining is evidence-driven rather than triggered by guesswork.

Deep dives and trade-offs

• Reproducibility and leakage preventionPin autonomous vehicle perception system data, feature, code, dependency, and model versions for every run. Use point-in-time joins and quarantine failed quality or privacy checks before training. Keep raw inputs and artifacts immutable so a result can be replayed after a dependency changes.
• Safe promotion and serving contractsSeparate autonomous vehicle perception system model registration from deployment and require signed artifacts plus schema compatibility. Use shadow traffic, canaries, rollback pointers, and per-version latency/error budgets. Return model version and feature freshness so clients can explain or reproduce a prediction.
• Drift, feedback, and costMeasure feature drift, prediction drift, label delay, and segment-level quality for autonomous vehicle perception system rather than only aggregate accuracy. Sample expensive inference and cap retraining concurrency with an explicit GPU or compute budget. Keep human corrections and delayed labels linked to the original prediction and model version.
• Batch versus online featuresPrefer a shared feature contract with batch backfills and a low-latency online serving path for decisions that need freshness. Two independently defined transformations create training-serving skew and hard-to-debug regressions.
• Synchronous versus asynchronous inferenceKeep interactive autonomous vehicle perception system inference synchronous within a strict budget and queue large or expensive jobs. A request path that waits for model loading, enrichment, or retraining turns downstream slowness into an outage.
• Global model versus segment modelsStart with one versioned model and add segment-specific models only when quality or policy evidence justifies the operational cost. Many simultaneously active versions multiply monitoring, rollback, and data-lineage burden.