Design a Multi-Modal AI Assistant — System Design Interview Practice

Interview prompt

Design a multimodal AI assistant that accepts text, images, audio, and video, extracts and fuses relevant representations, answers or generates content, and takes authorized actions with safe fallbacks.

• Separate original media, extracted transcripts and embeddings, conversation state, model policy, tool permissions, and outputs.
• Use modality-specific preprocessing and bounded fusion context, with asynchronous video/audio work and streaming text responses.
• Apply input, retrieval, action, and output safety controls; require confirmation for consequential or irreversible actions.
• Explain modality failure, latency budgets, privacy, prompt injection, media retention, and cost-aware model routing.

Requirements and scale assumptions

• Upload or stream modalities, transcribe and analyze media, answer questions, generate responses, and invoke authorized tools.
• Show citations or media evidence, partial progress, confidence, safety decisions, tool results, and conversation history.
• Support cancellation, media deletion, tenant isolation, consent, feedback, export, and model/version traceability.
• Target p95 text time to first token below one second while longer audio and video analysis reports progress asynchronously.
• Handle 50,000 concurrent sessions with modality-specific queues and per-tenant media and token quotas.
• Make uploads, turns, tool calls, moderation, and usage accounting idempotent and auditable.
• Fall back to text-only or a smaller model when a modality encoder, GPU pool, or external tool is unavailable.
• Support 50,000 concurrent sessions, 10,000 turns per second, and large audio/video uploads with varied durations.
• Partition by tenant, conversation, modality, and model; isolate long media jobs and high-volume assistants.
• Retain encrypted source media, consent, safety decisions, transcripts, and tool audit while bounding active context.

• Peak scale: Implement safety filters across all modalities — Capacity assumption that drives partitioning and backpressure.
• Latency target: Text response latency under 1 second — User-facing budget for the primary request or read path.
• Durable boundary: Committed before async — The source of truth is Accept text, image, audio, and video inputs; Reason across multiple modalities.
• Async boundary: At-least-once workers — Keep Use vision-language models for image understanding, Implement Whisper for speech-to-text, Use separate encoders per modality with fusion layers off the synchronous path.

Key entities

• DatasetVersiondatasetId, version, schemaHash, qualityStatus, lineage, createdAt

  Immutable multi modal ai assistant input version used for reproducible training, evaluation, or replay.

• FeatureSnapshotentityId, featureSetVersion, eventTime, values, sourceWatermarks

  Point-in-time multi modal ai assistant features with source watermarks so online and offline values can be compared.

• TrainingRunrunId, datasetVersion, codeVersion, metrics, artifactUri, status

  Audited multi modal ai assistant run that records data, code, dependency, and evaluation lineage.

• ModelVersionmodelId, version, stage, schema, qualityGates, endpoint

  A promotable multi modal ai assistant model version with rollout state, contract, and rollback metadata.

Data flow

1. 1. Register and validate training dataThe multi modal ai assistant gateway records an immutable dataset version, schema, lineage, quality status, and privacy disposition.
2. 2. Build point-in-time featuresFeature workers join multi modal ai assistant inputs using event-time watermarks, prevent leakage, and publish the same feature contract for training and serving.
3. 3. Train and evaluate asynchronouslyThe orchestrator schedules multi modal ai assistant runs with checkpointed artifacts, reproducible environments, and metrics tied to the exact input versions.
4. 4. Gate and serve a model versionA registry compares multi modal ai assistant quality, bias, safety, and compatibility gates before canary or production rollout with an immediate rollback pointer.
5. 5. Monitor drift and learn from feedbackOnline inference records latency, errors, drift, and delayed labels so multi modal ai assistant retraining is evidence-driven rather than triggered by guesswork.

Deep dives and trade-offs

• Reproducibility and leakage preventionPin multi modal ai assistant data, feature, code, dependency, and model versions for every run. Use point-in-time joins and quarantine failed quality or privacy checks before training. Keep raw inputs and artifacts immutable so a result can be replayed after a dependency changes.
• Safe promotion and serving contractsSeparate multi modal ai assistant model registration from deployment and require signed artifacts plus schema compatibility. Use shadow traffic, canaries, rollback pointers, and per-version latency/error budgets. Return model version and feature freshness so clients can explain or reproduce a prediction.
• Drift, feedback, and costMeasure feature drift, prediction drift, label delay, and segment-level quality for multi modal ai assistant rather than only aggregate accuracy. Sample expensive inference and cap retraining concurrency with an explicit GPU or compute budget. Keep human corrections and delayed labels linked to the original prediction and model version.
• Batch versus online featuresPrefer a shared feature contract with batch backfills and a low-latency online serving path for decisions that need freshness. Two independently defined transformations create training-serving skew and hard-to-debug regressions.
• Synchronous versus asynchronous inferenceKeep interactive multi modal ai assistant inference synchronous within a strict budget and queue large or expensive jobs. A request path that waits for model loading, enrichment, or retraining turns downstream slowness into an outage.
• Global model versus segment modelsStart with one versioned model and add segment-specific models only when quality or policy evidence justifies the operational cost. Many simultaneously active versions multiply monitoring, rollback, and data-lineage burden.