Create a System to Migrate Large Data to Google Cloud — System Design Interview Practice

Interview prompt

Design a resumable migration platform that transfers petabytes of data to Google Cloud, validates integrity and permissions, and minimizes application downtime during cutover.

• Treat source snapshots, change logs, checksums, and destination manifests as explicit migration phases with durable checkpoints.
• Copy files and database ranges in parallel, validate samples and full checksums, then drain changes before a reversible cutover.
• Protect source production traffic with bandwidth and read quotas and encrypt data in transit and at rest.
• Explain schema mapping, duplicate files, changed rows, failed chunks, rollback, retention, and audit evidence.

Requirements and scale assumptions

• Inventory sources, define destination mappings, transfer files and database partitions, and maintain incremental change capture.
• Expose per-object or per-range progress, checksum results, lag, throughput, validation failures, and cutover readiness.
• Support pause/resume, re-copy, schema transformation, access-policy mapping, final freeze, promotion, and rollback.
• Keep the final freeze and cutover within the agreed downtime window while bulk transfer runs for weeks beforehand.
• Transfer petabytes through partitioned resumable workers without saturating source systems or one hot prefix.
• Make chunks, change-log positions, validation reports, and promotion steps idempotent and auditable.
• Pause safely on source or destination faults and resume from checkpoints without skipping changed data.
• Migrate multiple petabytes of objects and tens of terabytes of databases across a multi-month program.
• Partition objects by prefix and databases by table or key range; isolate high-write tables and large files.
• Retain manifests, change logs, checksums, access mappings, and cutover evidence for replay and audit.

• Migration volume: Petabytes — Volume drives chunking, bandwidth limits, lifecycle tiers, and validation design.
• Cutover downtime: Planned freeze window — Bulk copy is asynchronous; only final change drain and promotion affect application availability.
• Durable boundary: Committed before async — The source remains authoritative until validated destination manifests are promoted atomically.
• Async boundary: At-least-once workers — Keep bulk copying, checksum validation, reconciliation, and reporting asynchronous.

Key entities

• SourcePartitionsourceId, partitionId, cursor, schemaVersion, watermark, status

  Replayable system to migrate large data to google cloud source evidence and ingestion cursor.

• SchemaVersiondatasetId, version, compatibility, owner, effectiveAt, status

  Governed system to migrate large data to google cloud contract used to validate producers and consumers.

• ProcessingRunrunId, inputWatermark, checkpoint, qualityStatus, codeVersion, status

  Checkpointed system to migrate large data to google cloud processing attempt with quality and lineage metadata.

• AnalyticalDatasetdatasetId, partition, watermark, schemaVersion, qualityStatus, location

  Curated system to migrate large data to google cloud serving partition with freshness and quality state.

Data flow

1. 1. Register sources and contractsThe system to migrate large data to google cloud catalog records owners, schemas, compatibility rules, retention, lineage, and partitioning before data is accepted.
2. 2. Ingest with backpressureConnectors checkpoint system to migrate large data to google cloud source cursors, validate schema and deduplication keys, and slow producers when downstream capacity is exhausted.
3. 3. Process event time with checkpointsStream or batch engines compute system to migrate large data to google cloud transformations using watermarks, late-data policy, state checkpoints, and deterministic code versions.
4. 4. Publish quality-gated datasetsOnly system to migrate large data to google cloud outputs that pass completeness, freshness, validity, and privacy checks become visible to analytical consumers.
5. 5. Serve, replay, and reconcileConsumers read bounded partitions with freshness metadata while operators replay failed system to migrate large data to google cloud ranges and compare output checksums.

Deep dives and trade-offs

• Schema evolution and data qualityVersion system to migrate large data to google cloud contracts and make compatibility rules explicit for every producer and consumer. Quarantine malformed partitions instead of poisoning the whole dataset. Track row counts, null rates, duplicates, distribution changes, and policy violations by partition.
• Watermarks, late data, and exactly-once effectsUse source cursors and event-time watermarks for system to migrate large data to google cloud progress, not wall-clock assumptions. Make checkpoints, output keys, and sink commits retry-safe under at-least-once delivery. Document how late events revise windows, aggregates, or snapshots.
• Replay, lineage, and costKeep immutable system to migrate large data to google cloud raw evidence and code or schema versions so failed outputs can be reproduced. Separate hot serving storage from cold retention and cap replay concurrency. Measure freshness, backlog, compute cost, storage growth, and quality-gate failure rate.
• Streaming versus batchUse streaming for freshness-critical system to migrate large data to google cloud paths and batch for backfills, compaction, and expensive recomputation. Forcing every workload into streaming makes state, replay, and cost harder to operate.
• Raw retention versus curated-only storageRetain enough immutable raw evidence for replay, audit, and correction, then tier or expire it according to policy. Without raw evidence, a bad transformation can require an unreproducible emergency fix.
• Central warehouse versus domain-owned datasetsCentralize governance and discovery while letting domain owners own contracts and quality signals. A single team owning every transformation becomes a delivery bottleneck and hides data ownership.