Build a Cognitive AI Application — System Design Interview Practice

Interview prompt

Design a multimodal conversational AI application that accepts text, images, and speech, retrieves grounded knowledge, understands intent, and answers users reliably at scale.

• Separate conversation state, tenant knowledge sources, document chunks, embeddings, prompts, and model outputs.
• Keep p95 response latency below two seconds with bounded retrieval, token budgets, streaming output, and model fallbacks.
• Ground answers in authorized knowledge and defend against prompt injection, data leakage, abuse, and hallucination.
• Make model calls, tool calls, safety filters, evaluation, and usage accounting observable and retry-safe.

Requirements and scale assumptions

• Create assistants, upload authorized knowledge, and configure supported text, image, speech, tools, and model policies.
• Accept a user turn, classify intent, retrieve relevant context, generate a cited response, and stream partial output.
• Store conversation history, feedback, safety decisions, token usage, traces, and evaluation results with tenant isolation.
• Target p95 time to first token below 500 ms and p95 completed response below 2 seconds for normal turns.
• Scale conversation traffic and embedding/index updates without unbounded prompt size or a hot tenant.
• Do not lose committed turns or audit events; make model retries and tool calls idempotent where possible.
• Degrade to a smaller model, cached retrieval, or an explicit unavailable response when dependencies fail.
• Support 100K concurrent conversations, 10K model requests per second, and millions of indexed documents.
• Partition conversation state by tenant and conversation; shard embeddings by collection and isolate large tenants.
• Bound history and retrieved context in prompts while retaining encrypted audit and evaluation data for replay.

• Peak model traffic: 10K requests/s — Drives model quotas, admission control, prompt budgets, and concurrency pools.
• Response latency: p95 <=2s — Completed response target; time to first token is separately tracked.
• Durable boundary: Committed before async — The source of truth is the committed conversation turn, assistant policy, and knowledge version.
• Async boundary: At-least-once workers — Keep indexing, evaluation, analytics, and long-running tool work off the synchronous response path.

Key entities

• DatasetVersiondatasetId, version, schemaHash, qualityStatus, lineage, createdAt

  Immutable cognitive ai application input version used for reproducible training, evaluation, or replay.

• FeatureSnapshotentityId, featureSetVersion, eventTime, values, sourceWatermarks

  Point-in-time cognitive ai application features with source watermarks so online and offline values can be compared.

• TrainingRunrunId, datasetVersion, codeVersion, metrics, artifactUri, status

  Audited cognitive ai application run that records data, code, dependency, and evaluation lineage.

• ModelVersionmodelId, version, stage, schema, qualityGates, endpoint

  A promotable cognitive ai application model version with rollout state, contract, and rollback metadata.

Data flow

1. 1. Register and validate training dataThe cognitive ai application gateway records an immutable dataset version, schema, lineage, quality status, and privacy disposition.
2. 2. Build point-in-time featuresFeature workers join cognitive ai application inputs using event-time watermarks, prevent leakage, and publish the same feature contract for training and serving.
3. 3. Train and evaluate asynchronouslyThe orchestrator schedules cognitive ai application runs with checkpointed artifacts, reproducible environments, and metrics tied to the exact input versions.
4. 4. Gate and serve a model versionA registry compares cognitive ai application quality, bias, safety, and compatibility gates before canary or production rollout with an immediate rollback pointer.
5. 5. Monitor drift and learn from feedbackOnline inference records latency, errors, drift, and delayed labels so cognitive ai application retraining is evidence-driven rather than triggered by guesswork.

Deep dives and trade-offs

• Reproducibility and leakage preventionPin cognitive ai application data, feature, code, dependency, and model versions for every run. Use point-in-time joins and quarantine failed quality or privacy checks before training. Keep raw inputs and artifacts immutable so a result can be replayed after a dependency changes.
• Safe promotion and serving contractsSeparate cognitive ai application model registration from deployment and require signed artifacts plus schema compatibility. Use shadow traffic, canaries, rollback pointers, and per-version latency/error budgets. Return model version and feature freshness so clients can explain or reproduce a prediction.
• Drift, feedback, and costMeasure feature drift, prediction drift, label delay, and segment-level quality for cognitive ai application rather than only aggregate accuracy. Sample expensive inference and cap retraining concurrency with an explicit GPU or compute budget. Keep human corrections and delayed labels linked to the original prediction and model version.
• Batch versus online featuresPrefer a shared feature contract with batch backfills and a low-latency online serving path for decisions that need freshness. Two independently defined transformations create training-serving skew and hard-to-debug regressions.
• Synchronous versus asynchronous inferenceKeep interactive cognitive ai application inference synchronous within a strict budget and queue large or expensive jobs. A request path that waits for model loading, enrichment, or retraining turns downstream slowness into an outage.
• Global model versus segment modelsStart with one versioned model and add segment-specific models only when quality or policy evidence justifies the operational cost. Many simultaneously active versions multiply monitoring, rollback, and data-lineage burden.