Design a Distributed Metrics Logging and Aggregation System — System Design Interview Practice

Interview prompt

Design a metrics and log aggregation platform that collects counters, gauges, histograms, traces, and structured logs from thousands of servers, supports alerting, and controls cardinality and retention cost.

• Define metric identity, timestamp, labels, units, and aggregation semantics separately from raw log and trace envelopes.
• Batch and compress at agents, enforce tenant and label budgets, shard ingestion by series, and downsample older data.
• Keep alert evaluation predictable and isolate it from exploratory queries and high-cardinality ingestion.
• Explain clock skew, duplicate samples, agent buffering, missing data, retention tiers, privacy, and alert correctness.

Requirements and scale assumptions

• Register sources and schemas, collect metric and log batches, query time ranges, aggregate series, and configure alerts.
• Support counters, gauges, histograms, exemplars, structured logs, retention policies, and downsampled rollups.
• Expose ingestion health, query results, alert state, missing-data indicators, cardinality warnings, and tenant usage.
• Target p95 ingestion acknowledgement below 1 second and query common dashboard ranges below 2 seconds.
• Handle billions of data points per day with agent batching, partitioned WALs, compression, and cardinality limits.
• Make samples and log batches deduplicable, preserve alert evaluation checkpoints, and distinguish missing from zero.
• Buffer at agents and serve recent local or downsampled data when remote storage or query workers are degraded.
• Collect 5 million samples per second, 2 billion points per day, and logs from 10,000 hosts.
• Partition by tenant, metric series, and time; isolate high-cardinality labels and noisy sources.
• Retain a short-resolution hot window and compressed rollups while archiving raw logs by policy.

• Peak scale: Handle billions of data points per day — Capacity assumption that drives partitioning and backpressure.
• Latency target: High write throughput — User-facing budget for the primary request or read path.
• Durable boundary: Committed before async — The source of truth is Collect metrics from thousands of servers; Support different metric types (counters, gauges, histograms).
• Async boundary: At-least-once workers — Keep Time-series database like Prometheus or InfluxDB, Data aggregation at collection time, Sampling for high-cardinality metrics off the synchronous path.

Key entities

• SourcePartitionsourceId, partitionId, cursor, schemaVersion, watermark, status

  Replayable distributed metrics logging and aggregation system source evidence and ingestion cursor.

• SchemaVersiondatasetId, version, compatibility, owner, effectiveAt, status

  Governed distributed metrics logging and aggregation system contract used to validate producers and consumers.

• ProcessingRunrunId, inputWatermark, checkpoint, qualityStatus, codeVersion, status

  Checkpointed distributed metrics logging and aggregation system processing attempt with quality and lineage metadata.

• AnalyticalDatasetdatasetId, partition, watermark, schemaVersion, qualityStatus, location

  Curated distributed metrics logging and aggregation system serving partition with freshness and quality state.

Data flow

1. 1. Register sources and contractsThe distributed metrics logging and aggregation system catalog records owners, schemas, compatibility rules, retention, lineage, and partitioning before data is accepted.
2. 2. Ingest with backpressureConnectors checkpoint distributed metrics logging and aggregation system source cursors, validate schema and deduplication keys, and slow producers when downstream capacity is exhausted.
3. 3. Process event time with checkpointsStream or batch engines compute distributed metrics logging and aggregation system transformations using watermarks, late-data policy, state checkpoints, and deterministic code versions.
4. 4. Publish quality-gated datasetsOnly distributed metrics logging and aggregation system outputs that pass completeness, freshness, validity, and privacy checks become visible to analytical consumers.
5. 5. Serve, replay, and reconcileConsumers read bounded partitions with freshness metadata while operators replay failed distributed metrics logging and aggregation system ranges and compare output checksums.

Deep dives and trade-offs

• Schema evolution and data qualityVersion distributed metrics logging and aggregation system contracts and make compatibility rules explicit for every producer and consumer. Quarantine malformed partitions instead of poisoning the whole dataset. Track row counts, null rates, duplicates, distribution changes, and policy violations by partition.
• Watermarks, late data, and exactly-once effectsUse source cursors and event-time watermarks for distributed metrics logging and aggregation system progress, not wall-clock assumptions. Make checkpoints, output keys, and sink commits retry-safe under at-least-once delivery. Document how late events revise windows, aggregates, or snapshots.
• Replay, lineage, and costKeep immutable distributed metrics logging and aggregation system raw evidence and code or schema versions so failed outputs can be reproduced. Separate hot serving storage from cold retention and cap replay concurrency. Measure freshness, backlog, compute cost, storage growth, and quality-gate failure rate.
• Streaming versus batchUse streaming for freshness-critical distributed metrics logging and aggregation system paths and batch for backfills, compaction, and expensive recomputation. Forcing every workload into streaming makes state, replay, and cost harder to operate.
• Raw retention versus curated-only storageRetain enough immutable raw evidence for replay, audit, and correction, then tier or expire it according to policy. Without raw evidence, a bad transformation can require an unreproducible emergency fix.
• Central warehouse versus domain-owned datasetsCentralize governance and discovery while letting domain owners own contracts and quality signals. A single team owning every transformation becomes a delivery bottleneck and hides data ownership.