Build a Scalable Data Lake Architecture — System Design Interview Practice

Interview prompt

Design a governed data lake that ingests batch and streaming sources, preserves raw structured/unstructured data, catalogs schemas and lineage, enables performant SQL, and serves trusted analytics.

• Define raw, validated, curated, and aggregate zones; dataset ownership, schema contracts, partitions, retention, lineage, and quality gates.
• Separate ingestion from compaction, cataloging, transformation, query engines, and BI serving; optimize file sizes and partition pruning.
• Handle late data, schema evolution, duplicates, backfills, sensitive fields, deletion requests, and exactly-once or replay semantics.
• Explain governance, cost controls, isolation, query correctness, disaster recovery, observability, and degraded query behavior.

Requirements and scale assumptions

• Ingest files, database changes, and streams into durable raw zones; validate, deduplicate, transform, catalog, and publish curated tables.
• Provide dataset discovery, schema/lineage views, SQL queries, materialized aggregates, BI access, freshness and quality status.
• Support role/row/column access, retention/deletion, replay/backfill, schema approval, failed-batch quarantine, and restore testing.
• Return common partition-pruned analytical queries with p95 under 10 seconds and publish freshness/quality metadata.
• Scale to petabytes and thousands of concurrent workloads without a single hot key or unbounded synchronous work.
• Do not lose committed state; make retries and duplicate events safe.
• Degrade safely when downstream workers, caches, or external dependencies fail.
• 5PB retained, 100TB daily ingest, and 10k concurrent analytical users
• Partition by the primary tenant, user, item, or geographic key and isolate hot partitions.
• Keep serving state bounded; retain raw events or durable records for replay and auditing.

• Peak scale: 5PB retained; 100TB/day ingest — Capacity assumption that drives partitioning and backpressure.
• Latency target: partitioned query p95 < 10s — User-facing budget for the primary request or read path.
• Durable boundary: Committed before async — Immutable raw objects and approved table manifests are authoritative; catalogs, indexes, and aggregates are derived.
• Async boundary: At-least-once workers — Keep S3 for data lake storage, AWS Glue for ETL and catalog, Athena for SQL queries off the synchronous path.

Key entities

• SourcePartitionsourceId, partitionId, cursor, schemaVersion, watermark, status

  Replayable scalable data lake architecture source evidence and ingestion cursor.

• SchemaVersiondatasetId, version, compatibility, owner, effectiveAt, status

  Governed scalable data lake architecture contract used to validate producers and consumers.

• ProcessingRunrunId, inputWatermark, checkpoint, qualityStatus, codeVersion, status

  Checkpointed scalable data lake architecture processing attempt with quality and lineage metadata.

• AnalyticalDatasetdatasetId, partition, watermark, schemaVersion, qualityStatus, location

  Curated scalable data lake architecture serving partition with freshness and quality state.

Data flow

1. 1. Register sources and contractsThe scalable data lake architecture catalog records owners, schemas, compatibility rules, retention, lineage, and partitioning before data is accepted.
2. 2. Ingest with backpressureConnectors checkpoint scalable data lake architecture source cursors, validate schema and deduplication keys, and slow producers when downstream capacity is exhausted.
3. 3. Process event time with checkpointsStream or batch engines compute scalable data lake architecture transformations using watermarks, late-data policy, state checkpoints, and deterministic code versions.
4. 4. Publish quality-gated datasetsOnly scalable data lake architecture outputs that pass completeness, freshness, validity, and privacy checks become visible to analytical consumers.
5. 5. Serve, replay, and reconcileConsumers read bounded partitions with freshness metadata while operators replay failed scalable data lake architecture ranges and compare output checksums.

Deep dives and trade-offs

• Schema evolution and data qualityVersion scalable data lake architecture contracts and make compatibility rules explicit for every producer and consumer. Quarantine malformed partitions instead of poisoning the whole dataset. Track row counts, null rates, duplicates, distribution changes, and policy violations by partition.
• Watermarks, late data, and exactly-once effectsUse source cursors and event-time watermarks for scalable data lake architecture progress, not wall-clock assumptions. Make checkpoints, output keys, and sink commits retry-safe under at-least-once delivery. Document how late events revise windows, aggregates, or snapshots.
• Replay, lineage, and costKeep immutable scalable data lake architecture raw evidence and code or schema versions so failed outputs can be reproduced. Separate hot serving storage from cold retention and cap replay concurrency. Measure freshness, backlog, compute cost, storage growth, and quality-gate failure rate.
• Streaming versus batchUse streaming for freshness-critical scalable data lake architecture paths and batch for backfills, compaction, and expensive recomputation. Forcing every workload into streaming makes state, replay, and cost harder to operate.
• Raw retention versus curated-only storageRetain enough immutable raw evidence for replay, audit, and correction, then tier or expire it according to policy. Without raw evidence, a bad transformation can require an unreproducible emergency fix.
• Central warehouse versus domain-owned datasetsCentralize governance and discovery while letting domain owners own contracts and quality signals. A single team owning every transformation becomes a delivery bottleneck and hides data ownership.