Design a Data Labeling and Annotation Platform — System Design Interview Practice

Interview prompt

Design a multimodal data-labeling platform for image, text, and audio tasks with collaborative workflows, consensus quality controls, active-learning prioritization, and exportable dataset versions.

• Define immutable source items, task schemas, annotation versions, assignments, consensus, adjudication, quality scores, and dataset snapshots.
• Support concurrent annotation without lost work, model-assisted suggestions, gold tasks, reviewer queues, disagreement handling, and active learning.
• Separate asset storage and task orchestration from browser collaboration, quality analytics, model training, and export pipelines.
• Explain tenant isolation, PII/access controls, moderation, auditability, retries, recovery, observability, and offline/degraded annotation.

Requirements and scale assumptions

• Create multimodal projects and schemas, import assets, assign tasks, annotate collaboratively, validate labels, adjudicate conflicts, and export versions.
• Provide pre-label suggestions, gold questions, reviewer workflows, throughput/quality dashboards, active-learning queues, and incremental exports.
• Support task lease expiry, annotation history, asset deletion, dataset rollback, annotator permissions, and recovery after browser/worker failure.
• Persist annotation changes with conflict-safe versions and update task/quality status quickly for active annotators.
• Scale to 10k concurrent annotators and 100M assets without a single hot key or unbounded synchronous work.
• Do not lose committed state; make retries and duplicate events safe.
• Degrade safely when downstream workers, caches, or external dependencies fail.
• 100M assets, 10k concurrent annotators, and multimodal projects
• Partition by the primary tenant, user, item, or geographic key and isolate hot partitions.
• Keep serving state bounded; retain raw events or durable records for replay and auditing.

• Peak scale: 100M assets; 10k annotators — Capacity assumption that drives partitioning and backpressure.
• Latency target: annotation save p95 < 500ms — User-facing budget for the primary request or read path.
• Durable boundary: Committed before async — Versioned annotations and dataset manifests are authoritative; task queues, suggestions, and dashboards are derived.
• Async boundary: At-least-once workers — Keep Use Label Studio or CVAT as open-source base, Implement active learning for intelligent sample selection, Use pre-trained models for label suggestions off the synchronous path.

Key entities

• SourcePartitionsourceId, partitionId, cursor, schemaVersion, watermark, status

  Replayable data labeling and annotation platform source evidence and ingestion cursor.

• SchemaVersiondatasetId, version, compatibility, owner, effectiveAt, status

  Governed data labeling and annotation platform contract used to validate producers and consumers.

• ProcessingRunrunId, inputWatermark, checkpoint, qualityStatus, codeVersion, status

  Checkpointed data labeling and annotation platform processing attempt with quality and lineage metadata.

• AnalyticalDatasetdatasetId, partition, watermark, schemaVersion, qualityStatus, location

  Curated data labeling and annotation platform serving partition with freshness and quality state.

Data flow

1. 1. Register sources and contractsThe data labeling and annotation platform catalog records owners, schemas, compatibility rules, retention, lineage, and partitioning before data is accepted.
2. 2. Ingest with backpressureConnectors checkpoint data labeling and annotation platform source cursors, validate schema and deduplication keys, and slow producers when downstream capacity is exhausted.
3. 3. Process event time with checkpointsStream or batch engines compute data labeling and annotation platform transformations using watermarks, late-data policy, state checkpoints, and deterministic code versions.
4. 4. Publish quality-gated datasetsOnly data labeling and annotation platform outputs that pass completeness, freshness, validity, and privacy checks become visible to analytical consumers.
5. 5. Serve, replay, and reconcileConsumers read bounded partitions with freshness metadata while operators replay failed data labeling and annotation platform ranges and compare output checksums.

Deep dives and trade-offs

• Schema evolution and data qualityVersion data labeling and annotation platform contracts and make compatibility rules explicit for every producer and consumer. Quarantine malformed partitions instead of poisoning the whole dataset. Track row counts, null rates, duplicates, distribution changes, and policy violations by partition.
• Watermarks, late data, and exactly-once effectsUse source cursors and event-time watermarks for data labeling and annotation platform progress, not wall-clock assumptions. Make checkpoints, output keys, and sink commits retry-safe under at-least-once delivery. Document how late events revise windows, aggregates, or snapshots.
• Replay, lineage, and costKeep immutable data labeling and annotation platform raw evidence and code or schema versions so failed outputs can be reproduced. Separate hot serving storage from cold retention and cap replay concurrency. Measure freshness, backlog, compute cost, storage growth, and quality-gate failure rate.
• Streaming versus batchUse streaming for freshness-critical data labeling and annotation platform paths and batch for backfills, compaction, and expensive recomputation. Forcing every workload into streaming makes state, replay, and cost harder to operate.
• Raw retention versus curated-only storageRetain enough immutable raw evidence for replay, audit, and correction, then tier or expire it according to policy. Without raw evidence, a bad transformation can require an unreproducible emergency fix.
• Central warehouse versus domain-owned datasetsCentralize governance and discovery while letting domain owners own contracts and quality signals. A single team owning every transformation becomes a delivery bottleneck and hides data ownership.