Design a Large Language Model Serving Platform — System Design Interview Practice

Interview prompt

Design an LLM inference platform that serves multiple model variants with streaming output, continuous batching, KV-cache management, GPU isolation, quotas, and safe rollout and fallback.

• Separate model and tokenizer artifacts, deployment policy, request queues, GPU workers, KV caches, and usage accounting.
• Route by model, context length, priority, and tenant while using continuous batching without allowing long prompts to starve short requests.
• Stream tokens with cancellation and backpressure, enforce prompt/output limits, and preserve the last healthy model during rollout.
• Explain quantization, autoscaling, cache eviction, multi-region routing, safety, privacy, and cost attribution.

Requirements and scale assumptions

• Register model versions, deploy GPU pools, submit chat or completion requests, stream tokens, cancel requests, and retrieve usage.
• Support model routing, prompt templates, safety policy, tenant quotas, priority, batch inference, and deployment health.
• Expose time to first token, inter-token latency, queue time, GPU utilization, cache hit rate, errors, and model version.
• Target p95 time to first token below 500 ms and bound output streaming latency for interactive requests.
• Handle 100,000 concurrent requests through model-specific queues and GPU pools with admission control.
• Make request accounting, cancellation, retries, and billing idempotent; do not duplicate external tool effects.
• Route to a smaller or cached model and preserve active streams when a model pool is degraded.
• Serve 100,000 concurrent requests, 10,000 tokens per second per pool, and dozens of model variants.
• Partition by model, region, tenant, and prompt-size class; isolate long-context and batch workloads.
• Retain signed model manifests, request metadata, safety decisions, and usage records while bounding KV caches.

• Peak scale: Handle concurrent inference requests — Capacity assumption that drives partitioning and backpressure.
• Latency target: Time-to-first-token under 500ms — User-facing budget for the primary request or read path.
• Durable boundary: Committed before async — The source of truth is Serve multiple LLM variants simultaneously; Handle concurrent inference requests.
• Async boundary: At-least-once workers — Keep Use vLLM or TensorRT-LLM for optimized inference, Implement KV-cache management for memory efficiency, Use continuous batching for higher throughput off the synchronous path.

Key entities

• DatasetVersiondatasetId, version, schemaHash, qualityStatus, lineage, createdAt

  Immutable large language model serving platform input version used for reproducible training, evaluation, or replay.

• FeatureSnapshotentityId, featureSetVersion, eventTime, values, sourceWatermarks

  Point-in-time large language model serving platform features with source watermarks so online and offline values can be compared.

• TrainingRunrunId, datasetVersion, codeVersion, metrics, artifactUri, status

  Audited large language model serving platform run that records data, code, dependency, and evaluation lineage.

• ModelVersionmodelId, version, stage, schema, qualityGates, endpoint

  A promotable large language model serving platform model version with rollout state, contract, and rollback metadata.

Data flow

1. 1. Register and validate training dataThe large language model serving platform gateway records an immutable dataset version, schema, lineage, quality status, and privacy disposition.
2. 2. Build point-in-time featuresFeature workers join large language model serving platform inputs using event-time watermarks, prevent leakage, and publish the same feature contract for training and serving.
3. 3. Train and evaluate asynchronouslyThe orchestrator schedules large language model serving platform runs with checkpointed artifacts, reproducible environments, and metrics tied to the exact input versions.
4. 4. Gate and serve a model versionA registry compares large language model serving platform quality, bias, safety, and compatibility gates before canary or production rollout with an immediate rollback pointer.
5. 5. Monitor drift and learn from feedbackOnline inference records latency, errors, drift, and delayed labels so large language model serving platform retraining is evidence-driven rather than triggered by guesswork.

Deep dives and trade-offs

• Reproducibility and leakage preventionPin large language model serving platform data, feature, code, dependency, and model versions for every run. Use point-in-time joins and quarantine failed quality or privacy checks before training. Keep raw inputs and artifacts immutable so a result can be replayed after a dependency changes.
• Safe promotion and serving contractsSeparate large language model serving platform model registration from deployment and require signed artifacts plus schema compatibility. Use shadow traffic, canaries, rollback pointers, and per-version latency/error budgets. Return model version and feature freshness so clients can explain or reproduce a prediction.
• Drift, feedback, and costMeasure feature drift, prediction drift, label delay, and segment-level quality for large language model serving platform rather than only aggregate accuracy. Sample expensive inference and cap retraining concurrency with an explicit GPU or compute budget. Keep human corrections and delayed labels linked to the original prediction and model version.
• Batch versus online featuresPrefer a shared feature contract with batch backfills and a low-latency online serving path for decisions that need freshness. Two independently defined transformations create training-serving skew and hard-to-debug regressions.
• Synchronous versus asynchronous inferenceKeep interactive large language model serving platform inference synchronous within a strict budget and queue large or expensive jobs. A request path that waits for model loading, enrichment, or retraining turns downstream slowness into an outage.
• Global model versus segment modelsStart with one versioned model and add segment-specific models only when quality or policy evidence justifies the operational cost. Many simultaneously active versions multiply monitoring, rollback, and data-lineage burden.