AI System Design Interview

How AI system design differs from classic system design

A classic design question asks you to build a deterministic system that scales. An AI design question keeps all of that and adds three things: the output is probabilistic, the quality depends on data that keeps changing, and the most expensive component, the model, behaves differently from ordinary services.

That means a complete answer covers more than boxes and arrows. You need a definition of quality, a path for data and models to be updated safely, a budget for latency and cost, and a plan for the moments when the model is wrong or unavailable.

Separate the offline path from the online path

The single most useful structure for any ML or LLM design is to split it into two flows that run on different schedules and have different requirements.

• Offline path: data ingestion, cleaning, labeling, feature computation, training or index building, evaluation, and registering a versioned model or index artifact. It is throughput-oriented and can take minutes to days.
• Online path: receiving a request, retrieving features or context, running inference, post-processing, returning a response, and logging inputs and outputs for feedback. It is latency-oriented and must respond in milliseconds to seconds.

Draw both, and draw the feedback loop that connects them: logged predictions and user outcomes flow back into the offline path as new training and evaluation data. Interviewers often probe whether you remembered that loop.

Define quality before you design

Ask what the model must produce and how anyone will know it is good. Without a metric, every architecture choice is unfalsifiable. Distinguish offline metrics, measured on a held-out evaluation set before release, from online metrics, measured on real traffic, usually through an A/B test.

Examples of how the metric depends on the product:

A retrieval-augmented generation (RAG) design

Retrieval-augmented generation grounds a language model in your own documents. The offline path splits documents into chunks, converts each chunk into an embedding vector, and stores the vectors in an index. The online path embeds the user's question, retrieves the most similar chunks, optionally reranks them, places them in the prompt, and asks the model to answer using that context, ideally with citations.

Each step contains a trade-off worth naming:

• Chunk size: small chunks are precise but lose context; large chunks keep context but dilute relevance and use more of the prompt.
• Number of retrieved chunks: more chunks raise recall but increase cost, latency, and the chance of distracting the model.
• Reranking: a second, more accurate model improves ordering at the price of extra latency and compute.
• Hybrid search: combining keyword search with vector search often handles exact terms, such as product codes, better than vectors alone.
• Freshness: decide how quickly changed documents must appear in the index, and design incremental re-indexing rather than rebuilding everything.

Budget latency, quality, and cost together

In most AI products the model call dominates both latency and cost, so the design question becomes where you can avoid, shrink, or hide it. Name the lever and its price:

Evaluate before release and monitor after

Model quality changes after launch because the world changes. Input data drifts, user behavior shifts, and upstream pipelines break silently. A strong design treats evaluation as an ongoing system, not a one-time test.

Before release, run the candidate against a fixed evaluation set and compare it with the current model. Release gradually, using shadow mode to run it without serving its output, then a canary on a small share of traffic, then a wider A/B test.

• Monitor input distributions to detect data drift.
• Sample and review outputs, using automated checks and human review, to catch quality regressions.
• Track operational metrics: latency percentiles, error rates, token or compute usage, and cost.
• Log enough to reproduce a bad answer: the model version, the retrieved context, and the prompt.

Plan for degradation and rollback

Interviewers like to ask what happens when the model, the vector index, the feature store, or a third-party provider is unavailable. Have an answer for each, in order from best to worst experience.

• Fall back to a smaller model, a cached answer, or a simple rule-based or keyword result.
• Return a clear partial result or an honest error instead of a confident wrong answer.
• Keep previous model and index versions deployed so you can roll back quickly, and make the switch a configuration change.
• Provide a kill switch for a misbehaving feature, and a human review path where mistakes are costly.

Safety, privacy, and access control

Production AI systems need a few non-negotiable controls, and mentioning them signals maturity. Apply access control before retrieval, so a user can only receive context from documents they are permitted to see; filtering afterward risks leaking restricted content into the prompt.

Treat retrieved text and user input as untrusted. A document or web page can contain instructions that try to hijack the model, a risk called prompt injection, so keep untrusted content separated from system instructions and limit what actions the model can trigger. Minimize and mask personal data in logs and training sets, and define retention rules.

A worked estimate: semantic search

Suppose the prompt is a semantic search over one million support articles, using 768-dimensional embeddings stored as 32-bit floats.

Each vector is 768 times 4 bytes, which is 3,072 bytes. One million vectors is about 3 GB of raw vector data. Indexes add overhead, so plan for a few multiples of that, but the result is small enough to hold in memory on a single node. At 50 queries per second, a replicated pair of index nodes is comfortable, and the model that creates query embeddings is likely the larger latency cost than the vector lookup itself.

The numbers lead directly to design decisions: no sharding is needed at this size, replication is for availability rather than capacity, and the effort belongs in chunking, reranking, and evaluation quality rather than infrastructure.

Common mistakes

• Designing only the serving path and forgetting how data and models are produced and updated.
• Never defining a quality metric, so no design can be judged better or worse.
• Ignoring cost. A design that is accurate but costs too much per request is not viable.
• No fallback, so a model outage becomes a full product outage.
• Evaluating once at launch and never monitoring drift afterward.
• Putting user data in prompts or logs without considering privacy and access control.