Design a Real-time AI Translation and Localization Service — System Design Interview Practice

Interview prompt

Design a real-time translation and localization service for text, speech, and documents across 100+ languages, with terminology control, context, quality estimation, and streaming output.

• Separate source text or media, locale and terminology policy, model versions, translation memory, quality scores, and published locale assets.
• Route by language pair, domain, and latency class, stream speech segments, and keep terminology and protected names consistent.
• Make translation requests idempotent, preserve segment boundaries and timestamps, and support human review for critical content.
• Explain context windows, code switching, partial audio, glossary updates, privacy, fallback models, and cost controls.

Requirements and scale assumptions

• Translate text, speech, and documents, apply locale formatting and terminology, stream partial results, and return quality estimates.
• Support language detection, glossary and translation-memory lookup, speaker timestamps, document layout, and human corrections.
• Expose model and glossary versions, segment status, confidence, latency, usage, retention, and tenant access controls.
• Target p95 text translation latency below 200 ms for short segments and publish speech tokens with bounded lag.
• Serve 100,000 segment requests per second across 100+ language pairs with language-specific pools and quotas.
• Make segment requests, glossary publication, document jobs, and billing idempotent with stable source hashes.
• Fall back to a general model or queued document translation when a language-specific model or terminology service fails.
• Support 100+ languages, 10 million text segments per hour, and 100,000 concurrent speech or document sessions.
• Partition by tenant, language pair, domain, and session; isolate rare language pairs and long documents.
• Retain encrypted source hashes, segment outputs, glossary versions, quality scores, and human corrections under policy.

• Peak scale: Handle context-aware translations — Capacity assumption that drives partitioning and backpressure.
• Latency target: Text translation latency under 200ms — User-facing budget for the primary request or read path.
• Durable boundary: Committed before async — The source of truth is Translate text across 100+ language pairs; Provide real-time speech-to-speech translation.
• Async boundary: At-least-once workers — Keep Use transformer-based NMT models (mBART, NLLB), Implement streaming translation for speech, Use terminology databases for domain adaptation off the synchronous path.

Key entities

• DatasetVersiondatasetId, version, schemaHash, qualityStatus, lineage, createdAt

  Immutable real time ai translation and localization service input version used for reproducible training, evaluation, or replay.

• FeatureSnapshotentityId, featureSetVersion, eventTime, values, sourceWatermarks

  Point-in-time real time ai translation and localization service features with source watermarks so online and offline values can be compared.

• TrainingRunrunId, datasetVersion, codeVersion, metrics, artifactUri, status

  Audited real time ai translation and localization service run that records data, code, dependency, and evaluation lineage.

• ModelVersionmodelId, version, stage, schema, qualityGates, endpoint

  A promotable real time ai translation and localization service model version with rollout state, contract, and rollback metadata.

Data flow

1. 1. Register and validate training dataThe real time ai translation and localization service gateway records an immutable dataset version, schema, lineage, quality status, and privacy disposition.
2. 2. Build point-in-time featuresFeature workers join real time ai translation and localization service inputs using event-time watermarks, prevent leakage, and publish the same feature contract for training and serving.
3. 3. Train and evaluate asynchronouslyThe orchestrator schedules real time ai translation and localization service runs with checkpointed artifacts, reproducible environments, and metrics tied to the exact input versions.
4. 4. Gate and serve a model versionA registry compares real time ai translation and localization service quality, bias, safety, and compatibility gates before canary or production rollout with an immediate rollback pointer.
5. 5. Monitor drift and learn from feedbackOnline inference records latency, errors, drift, and delayed labels so real time ai translation and localization service retraining is evidence-driven rather than triggered by guesswork.

Deep dives and trade-offs

• Reproducibility and leakage preventionPin real time ai translation and localization service data, feature, code, dependency, and model versions for every run. Use point-in-time joins and quarantine failed quality or privacy checks before training. Keep raw inputs and artifacts immutable so a result can be replayed after a dependency changes.
• Safe promotion and serving contractsSeparate real time ai translation and localization service model registration from deployment and require signed artifacts plus schema compatibility. Use shadow traffic, canaries, rollback pointers, and per-version latency/error budgets. Return model version and feature freshness so clients can explain or reproduce a prediction.
• Drift, feedback, and costMeasure feature drift, prediction drift, label delay, and segment-level quality for real time ai translation and localization service rather than only aggregate accuracy. Sample expensive inference and cap retraining concurrency with an explicit GPU or compute budget. Keep human corrections and delayed labels linked to the original prediction and model version.
• Batch versus online featuresPrefer a shared feature contract with batch backfills and a low-latency online serving path for decisions that need freshness. Two independently defined transformations create training-serving skew and hard-to-debug regressions.
• Synchronous versus asynchronous inferenceKeep interactive real time ai translation and localization service inference synchronous within a strict budget and queue large or expensive jobs. A request path that waits for model loading, enrichment, or retraining turns downstream slowness into an outage.
• Global model versus segment modelsStart with one versioned model and add segment-specific models only when quality or policy evidence justifies the operational cost. Many simultaneously active versions multiply monitoring, rollback, and data-lineage burden.