Design a Distributed Tracing System — System Design Interview Practice

Interview prompt

Design a distributed tracing platform that propagates trace context across services, collects spans with low application overhead, samples intelligently, and supports fast latency and dependency analysis.

• Define trace and span identity, parentage, propagation, status, resource attributes, links, and clock-skew handling.
• Buffer and batch spans at agents, apply head or tail sampling, cap high-cardinality attributes, and partition by trace ID.
• Keep instrumentation non-blocking and preserve enough sampled evidence to diagnose tail latency and errors.
• Explain partial traces, collector overload, late spans, retention, tenant privacy, indexing, and query cost.

Requirements and scale assumptions

• Propagate context, create spans, collect batches, sample traces, store searchable data, and query a trace or service dependency.
• Show waterfall timing, errors, exemplars, service maps, latency percentiles, sampling decisions, and incomplete-span warnings.
• Support tenant access, PII redaction, retention policies, trace deletion, collector health, and export to incident tools.
• Keep synchronous instrumentation overhead below 1% and target p95 query latency below 2 seconds for recent traces.
• Ingest 10 billion spans per day with agent batching, sampling, compression, and trace-ID partitioning.
• Make collector retries and batch ingestion deduplicable while distinguishing incomplete traces from healthy short traces.
• Drop or sample low-value spans under pressure but preserve error, latency-tail, and policy-required traces.
• Collect 10 billion spans per day from 100,000 services with bursts during incidents.
• Partition by tenant, trace ID, and event time; isolate high-cardinality services and incident surges.
• Retain recent indexed spans and lower-cost raw archives under tenant-specific retention and privacy policy.

• Peak scale: Handle billions of spans per day — Capacity assumption that drives partitioning and backpressure.
• Latency target: Low overhead on application performance — User-facing budget for the primary request or read path.
• Durable boundary: Committed before async — The source of truth is Trace requests across distributed services; Capture timing and latency information.
• Async boundary: At-least-once workers — Keep Trace ID and span ID propagation, Sampling strategies (head-based, tail-based), Time-series storage for spans off the synchronous path.

Key entities

• SourcePartitionsourceId, partitionId, cursor, schemaVersion, watermark, status

  Replayable distributed tracing system source evidence and ingestion cursor.

• SchemaVersiondatasetId, version, compatibility, owner, effectiveAt, status

  Governed distributed tracing system contract used to validate producers and consumers.

• ProcessingRunrunId, inputWatermark, checkpoint, qualityStatus, codeVersion, status

  Checkpointed distributed tracing system processing attempt with quality and lineage metadata.

• AnalyticalDatasetdatasetId, partition, watermark, schemaVersion, qualityStatus, location

  Curated distributed tracing system serving partition with freshness and quality state.

Data flow

1. 1. Register sources and contractsThe distributed tracing system catalog records owners, schemas, compatibility rules, retention, lineage, and partitioning before data is accepted.
2. 2. Ingest with backpressureConnectors checkpoint distributed tracing system source cursors, validate schema and deduplication keys, and slow producers when downstream capacity is exhausted.
3. 3. Process event time with checkpointsStream or batch engines compute distributed tracing system transformations using watermarks, late-data policy, state checkpoints, and deterministic code versions.
4. 4. Publish quality-gated datasetsOnly distributed tracing system outputs that pass completeness, freshness, validity, and privacy checks become visible to analytical consumers.
5. 5. Serve, replay, and reconcileConsumers read bounded partitions with freshness metadata while operators replay failed distributed tracing system ranges and compare output checksums.

Deep dives and trade-offs

• Schema evolution and data qualityVersion distributed tracing system contracts and make compatibility rules explicit for every producer and consumer. Quarantine malformed partitions instead of poisoning the whole dataset. Track row counts, null rates, duplicates, distribution changes, and policy violations by partition.
• Watermarks, late data, and exactly-once effectsUse source cursors and event-time watermarks for distributed tracing system progress, not wall-clock assumptions. Make checkpoints, output keys, and sink commits retry-safe under at-least-once delivery. Document how late events revise windows, aggregates, or snapshots.
• Replay, lineage, and costKeep immutable distributed tracing system raw evidence and code or schema versions so failed outputs can be reproduced. Separate hot serving storage from cold retention and cap replay concurrency. Measure freshness, backlog, compute cost, storage growth, and quality-gate failure rate.
• Streaming versus batchUse streaming for freshness-critical distributed tracing system paths and batch for backfills, compaction, and expensive recomputation. Forcing every workload into streaming makes state, replay, and cost harder to operate.
• Raw retention versus curated-only storageRetain enough immutable raw evidence for replay, audit, and correction, then tier or expire it according to policy. Without raw evidence, a bad transformation can require an unreproducible emergency fix.
• Central warehouse versus domain-owned datasetsCentralize governance and discovery while letting domain owners own contracts and quality signals. A single team owning every transformation becomes a delivery bottleneck and hides data ownership.