Design Google Analytics - User Analytics Dashboard and Pipeline — System Design Interview Practice

Interview prompt

Design a privacy-aware product analytics pipeline that collects user events, computes near-real-time and historical reports, supports flexible dimensions, and remains cost-efficient at billions of events per day.

• Define an event schema, identity stitching, consent, bot filtering, deduplication, late-arrival policy, retention, and tenant isolation.
• Separate immutable raw events from streaming rollups and warehouse queries; support new dimensions without rewriting the ingestion contract.
• Explain approximate distinct counts, attribution windows, backfills, replay, schema evolution, dashboard freshness, and cost controls.
• Cover access control, deletion propagation, sampling, data quality, observability, and degraded reporting.

Requirements and scale assumptions

• Collect page views, clicks, sessions, conversions, and custom events from web/mobile clients with validation and retry-safe ingestion.
• Provide funnels, cohorts, retention, paths, real-time counters, segmentation, export, and scheduled reports with freshness metadata.
• Support consent withdrawal, user deletion, schema versions, replay/backfill, correction of late events, and tenant-scoped access.
• Publish common real-time aggregates within one minute while retaining raw-event durability for exact historical recomputation.
• Scale to billions of events per day and high-cardinality tenants without a single hot key or unbounded synchronous work.
• Do not lose committed state; make retries and duplicate events safe.
• Degrade safely when downstream workers, caches, or external dependencies fail.
• 5B events/day, 100k events/second peak, and 100k dashboard users
• Partition by the primary tenant, user, item, or geographic key and isolate hot partitions.
• Keep serving state bounded; retain raw events or durable records for replay and auditing.

• Peak scale: 5B events/day; 100k events/s peak — Capacity assumption that drives partitioning and backpressure.
• Latency target: common rollups < 1m; exact backfill async — User-facing budget for the primary request or read path.
• Durable boundary: Committed before async — Consent-checked immutable events are authoritative; rollups, cohorts, and dashboard caches are derived.
• Async boundary: At-least-once workers — Keep Lambda architecture (batch + stream), Data warehouse for historical analysis, Stream processing for real-time metrics off the synchronous path.

Key entities

• SourcePartitionsourceId, partitionId, cursor, schemaVersion, watermark, status

  Replayable google analytics user analytics dashboard and pipeline source evidence and ingestion cursor.

• SchemaVersiondatasetId, version, compatibility, owner, effectiveAt, status

  Governed google analytics user analytics dashboard and pipeline contract used to validate producers and consumers.

• ProcessingRunrunId, inputWatermark, checkpoint, qualityStatus, codeVersion, status

  Checkpointed google analytics user analytics dashboard and pipeline processing attempt with quality and lineage metadata.

• AnalyticalDatasetdatasetId, partition, watermark, schemaVersion, qualityStatus, location

  Curated google analytics user analytics dashboard and pipeline serving partition with freshness and quality state.

Data flow

1. 1. Register sources and contractsThe google analytics user analytics dashboard and pipeline catalog records owners, schemas, compatibility rules, retention, lineage, and partitioning before data is accepted.
2. 2. Ingest with backpressureConnectors checkpoint google analytics user analytics dashboard and pipeline source cursors, validate schema and deduplication keys, and slow producers when downstream capacity is exhausted.
3. 3. Process event time with checkpointsStream or batch engines compute google analytics user analytics dashboard and pipeline transformations using watermarks, late-data policy, state checkpoints, and deterministic code versions.
4. 4. Publish quality-gated datasetsOnly google analytics user analytics dashboard and pipeline outputs that pass completeness, freshness, validity, and privacy checks become visible to analytical consumers.
5. 5. Serve, replay, and reconcileConsumers read bounded partitions with freshness metadata while operators replay failed google analytics user analytics dashboard and pipeline ranges and compare output checksums.

Deep dives and trade-offs

• Schema evolution and data qualityVersion google analytics user analytics dashboard and pipeline contracts and make compatibility rules explicit for every producer and consumer. Quarantine malformed partitions instead of poisoning the whole dataset. Track row counts, null rates, duplicates, distribution changes, and policy violations by partition.
• Watermarks, late data, and exactly-once effectsUse source cursors and event-time watermarks for google analytics user analytics dashboard and pipeline progress, not wall-clock assumptions. Make checkpoints, output keys, and sink commits retry-safe under at-least-once delivery. Document how late events revise windows, aggregates, or snapshots.
• Replay, lineage, and costKeep immutable google analytics user analytics dashboard and pipeline raw evidence and code or schema versions so failed outputs can be reproduced. Separate hot serving storage from cold retention and cap replay concurrency. Measure freshness, backlog, compute cost, storage growth, and quality-gate failure rate.
• Streaming versus batchUse streaming for freshness-critical google analytics user analytics dashboard and pipeline paths and batch for backfills, compaction, and expensive recomputation. Forcing every workload into streaming makes state, replay, and cost harder to operate.
• Raw retention versus curated-only storageRetain enough immutable raw evidence for replay, audit, and correction, then tier or expire it according to policy. Without raw evidence, a bad transformation can require an unreproducible emergency fix.
• Central warehouse versus domain-owned datasetsCentralize governance and discovery while letting domain owners own contracts and quality signals. A single team owning every transformation becomes a delivery bottleneck and hides data ownership.