Design a Real-time AI Content Moderation System — System Design Interview Practice

Interview prompt

Design a real-time multimodal moderation platform that classifies text, images, and video across languages, applies policy decisions quickly, and routes uncertain or high-impact cases to human review.

• Keep submitted content and policy version authoritative while deriving model scores, decisions, reviewer queues, and appeals.
• Use a fast rules-and-model cascade for text and images, with asynchronous frame sampling for video and language-specific thresholds.
• Make decisions explainable enough for appeals, prevent model updates from changing historical evidence, and fail closed for severe risks.
• Explain false positives, adversarial content, reviewer safety, privacy, regional policy, model rollout, and deletion.

Requirements and scale assumptions

• Submit content for pre- or post-publication checks, classify policy categories, quarantine risky content, and return a versioned decision.
• Route low-confidence cases to reviewers, support appeals and overrides, and expose evidence, confidence, policy, and model versions.
• Support language and region policies, user reports, deletion, audit export, reviewer permissions, and model evaluation feedback.
• Target p95 text moderation below 200 ms and make high-risk actions deterministic even when deeper video analysis is pending.
• Process 100,000 pieces per second with modality and tenant queues that isolate viral content and large videos.
• Make content submission, scoring, quarantine, review assignment, and decision publication idempotent.
• Fall back to strict rules and quarantine for severe categories when model workers or feature services are unavailable.
• Moderate 100 million text items, 10 million images, and 1 million short videos per day across many languages.
• Partition by tenant, content ID, modality, language, and policy region; isolate viral items and review queues.
• Retain redacted content references, policy decisions, model versions, reviewer actions, appeals, and audit evidence.

• Peak scale: 100M items/day; 100k items/s peak — Capacity assumption that drives partitioning and backpressure.
• Latency target: text p95 < 200ms; severe risks fail closed — User-facing budget for the primary request or read path.
• Durable boundary: Committed before async — Submitted content references, policy versions, and published decisions are authoritative; scores and queues are derived.
• Async boundary: At-least-once workers — Keep Use multi-modal models for text and image classification, Implement cascade filtering (fast rules then ML models), Use embedding similarity for known harmful content off the synchronous path.

Key entities

• DatasetVersiondatasetId, version, schemaHash, qualityStatus, lineage, createdAt

  Immutable real time ai content moderation system input version used for reproducible training, evaluation, or replay.

• FeatureSnapshotentityId, featureSetVersion, eventTime, values, sourceWatermarks

  Point-in-time real time ai content moderation system features with source watermarks so online and offline values can be compared.

• TrainingRunrunId, datasetVersion, codeVersion, metrics, artifactUri, status

  Audited real time ai content moderation system run that records data, code, dependency, and evaluation lineage.

• ModelVersionmodelId, version, stage, schema, qualityGates, endpoint

  A promotable real time ai content moderation system model version with rollout state, contract, and rollback metadata.

Data flow

1. 1. Register and validate training dataThe real time ai content moderation system gateway records an immutable dataset version, schema, lineage, quality status, and privacy disposition.
2. 2. Build point-in-time featuresFeature workers join real time ai content moderation system inputs using event-time watermarks, prevent leakage, and publish the same feature contract for training and serving.
3. 3. Train and evaluate asynchronouslyThe orchestrator schedules real time ai content moderation system runs with checkpointed artifacts, reproducible environments, and metrics tied to the exact input versions.
4. 4. Gate and serve a model versionA registry compares real time ai content moderation system quality, bias, safety, and compatibility gates before canary or production rollout with an immediate rollback pointer.
5. 5. Monitor drift and learn from feedbackOnline inference records latency, errors, drift, and delayed labels so real time ai content moderation system retraining is evidence-driven rather than triggered by guesswork.

Deep dives and trade-offs

• Reproducibility and leakage preventionPin real time ai content moderation system data, feature, code, dependency, and model versions for every run. Use point-in-time joins and quarantine failed quality or privacy checks before training. Keep raw inputs and artifacts immutable so a result can be replayed after a dependency changes.
• Safe promotion and serving contractsSeparate real time ai content moderation system model registration from deployment and require signed artifacts plus schema compatibility. Use shadow traffic, canaries, rollback pointers, and per-version latency/error budgets. Return model version and feature freshness so clients can explain or reproduce a prediction.
• Drift, feedback, and costMeasure feature drift, prediction drift, label delay, and segment-level quality for real time ai content moderation system rather than only aggregate accuracy. Sample expensive inference and cap retraining concurrency with an explicit GPU or compute budget. Keep human corrections and delayed labels linked to the original prediction and model version.
• Batch versus online featuresPrefer a shared feature contract with batch backfills and a low-latency online serving path for decisions that need freshness. Two independently defined transformations create training-serving skew and hard-to-debug regressions.
• Synchronous versus asynchronous inferenceKeep interactive real time ai content moderation system inference synchronous within a strict budget and queue large or expensive jobs. A request path that waits for model loading, enrichment, or retraining turns downstream slowness into an outage.
• Global model versus segment modelsStart with one versioned model and add segment-specific models only when quality or policy evidence justifies the operational cost. Many simultaneously active versions multiply monitoring, rollback, and data-lineage burden.