Build a Real-time Analytics Dashboard — System Design Interview Practice

Interview prompt

Design a real-time analytics dashboard that ingests streaming events, maintains queryable aggregates, refreshes visualizations automatically, and controls analytical cost and freshness.

• Define event schema, dimensions, windows, late-data policy, aggregate semantics, dashboard freshness, and tenant/query access boundaries.
• Separate streaming rollups from warehouse history and interactive queries; use bounded state, pre-aggregation, caching, and query quotas.
• Handle duplicates, corrections, schema evolution, dashboard refresh storms, backfills, and approximate versus exact metrics.
• Explain data quality, privacy/deletion, cost controls, replay, observability, and stale/local dashboard fallback.

Requirements and scale assumptions

• Ingest validated events, update windowed aggregates, persist raw data, and expose SQL-backed dashboard datasets with freshness metadata.
• Provide configurable charts, filters, drill-down, alerts, scheduled refresh, saved queries, exports, and tenant-scoped sharing.
• Support corrections/backfills, retention/deletion, query cancellation, replay, dashboard versioning, and recovery after stream or warehouse failure.
• Meet p95 common dashboard query latency under five seconds and show aggregate watermark/freshness for every panel.
• Serve 100k concurrent viewers and billions of events per day without a single hot key or unbounded synchronous work.
• Do not lose committed state; make retries and duplicate events safe.
• Degrade safely when downstream workers, caches, or external dependencies fail.
• 1B events/day, 100k dashboard viewers, and 10k saved reports
• Partition by the primary tenant, user, item, or geographic key and isolate hot partitions.
• Keep serving state bounded; retain raw events or durable records for replay and auditing.

• Peak scale: 1B events/day; 100k viewers — Capacity assumption that drives partitioning and backpressure.
• Latency target: dashboard p95 < 5s; freshness visible — User-facing budget for the primary request or read path.
• Durable boundary: Committed before async — Immutable events and versioned aggregate definitions are authoritative; panels and caches are derived.
• Async boundary: At-least-once workers — Keep BigQuery for analytics warehouse, Pub/Sub for event streaming, Data Studio for visualization off the synchronous path.

Key entities

• SourcePartitionsourceId, partitionId, cursor, schemaVersion, watermark, status

  Replayable real time analytics dashboard source evidence and ingestion cursor.

• SchemaVersiondatasetId, version, compatibility, owner, effectiveAt, status

  Governed real time analytics dashboard contract used to validate producers and consumers.

• ProcessingRunrunId, inputWatermark, checkpoint, qualityStatus, codeVersion, status

  Checkpointed real time analytics dashboard processing attempt with quality and lineage metadata.

• AnalyticalDatasetdatasetId, partition, watermark, schemaVersion, qualityStatus, location

  Curated real time analytics dashboard serving partition with freshness and quality state.

Data flow

1. 1. Register sources and contractsThe real time analytics dashboard catalog records owners, schemas, compatibility rules, retention, lineage, and partitioning before data is accepted.
2. 2. Ingest with backpressureConnectors checkpoint real time analytics dashboard source cursors, validate schema and deduplication keys, and slow producers when downstream capacity is exhausted.
3. 3. Process event time with checkpointsStream or batch engines compute real time analytics dashboard transformations using watermarks, late-data policy, state checkpoints, and deterministic code versions.
4. 4. Publish quality-gated datasetsOnly real time analytics dashboard outputs that pass completeness, freshness, validity, and privacy checks become visible to analytical consumers.
5. 5. Serve, replay, and reconcileConsumers read bounded partitions with freshness metadata while operators replay failed real time analytics dashboard ranges and compare output checksums.

Deep dives and trade-offs

• Schema evolution and data qualityVersion real time analytics dashboard contracts and make compatibility rules explicit for every producer and consumer. Quarantine malformed partitions instead of poisoning the whole dataset. Track row counts, null rates, duplicates, distribution changes, and policy violations by partition.
• Watermarks, late data, and exactly-once effectsUse source cursors and event-time watermarks for real time analytics dashboard progress, not wall-clock assumptions. Make checkpoints, output keys, and sink commits retry-safe under at-least-once delivery. Document how late events revise windows, aggregates, or snapshots.
• Replay, lineage, and costKeep immutable real time analytics dashboard raw evidence and code or schema versions so failed outputs can be reproduced. Separate hot serving storage from cold retention and cap replay concurrency. Measure freshness, backlog, compute cost, storage growth, and quality-gate failure rate.
• Streaming versus batchUse streaming for freshness-critical real time analytics dashboard paths and batch for backfills, compaction, and expensive recomputation. Forcing every workload into streaming makes state, replay, and cost harder to operate.
• Raw retention versus curated-only storageRetain enough immutable raw evidence for replay, audit, and correction, then tier or expire it according to policy. Without raw evidence, a bad transformation can require an unreproducible emergency fix.
• Central warehouse versus domain-owned datasetsCentralize governance and discovery while letting domain owners own contracts and quality signals. A single team owning every transformation becomes a delivery bottleneck and hides data ownership.