Design a Multi-Cloud Data Sync Solution — System Design Interview Practice

Interview prompt

Design a multi-cloud synchronization platform that detects changes, transfers objects and records across three providers, resolves conflicts under an explicit ownership policy, and reports freshness and integrity.

• Define source ownership, object/version identity, deletion semantics, conflict resolution, and per-dataset consistency contracts.
• Use manifests, checksums, change logs, compression, batching, and resumable transfers while respecting provider quotas.
• Make replication idempotent and preserve a durable sync ledger so out-of-order notifications and retries are safe.
• Explain split-brain writes, provider outage, encryption, residency, cost, backfill, and repair reconciliation.

Requirements and scale assumptions

• Register datasets and providers, discover changes, transfer versions, apply deletes, resolve conflicts, and verify checksums.
• Expose per-provider watermarks, lag, transfer progress, conflict decisions, checksum failures, and repair runs.
• Support tenant-scoped credentials, key rotation, retention, pause/resume, backfill, export, and audit history.
• Propagate ordinary changes within five minutes while allowing large backfills to run with separate quotas.
• Synchronize petabyte-scale objects and billions of records without one hot prefix or provider API limit blocking all datasets.
• Never acknowledge a copy without checksum and version evidence; make notifications, transfers, and conflict writes idempotent.
• Pause a failed provider and continue healthy replication paths with clear freshness and conflict status.
• Sync 10 PB of objects and 100 billion records across three providers with uneven network and egress costs.
• Partition by tenant, dataset, provider, and object prefix; isolate high-change datasets and hot notification streams.
• Retain manifests, versions, checksums, conflict decisions, and audit events while bounding active transfer state.

• Peak scale: Optimize transfer costs — Capacity assumption that drives partitioning and backpressure.
• Latency target: Sync latency under 5 minutes — User-facing budget for the primary request or read path.
• Durable boundary: Committed before async — The source of truth is Sync data across three cloud providers; Detect and resolve conflicts.
• Async boundary: At-least-once workers — Keep Use native sync tools (DataSync, AzCopy, gsutil), Implement change detection (checksums, timestamps), Use event notifications for triggers off the synchronous path.

Key entities

• SourcePartitionsourceId, partitionId, cursor, schemaVersion, watermark, status

  Replayable multi cloud data sync solution source evidence and ingestion cursor.

• SchemaVersiondatasetId, version, compatibility, owner, effectiveAt, status

  Governed multi cloud data sync solution contract used to validate producers and consumers.

• ProcessingRunrunId, inputWatermark, checkpoint, qualityStatus, codeVersion, status

  Checkpointed multi cloud data sync solution processing attempt with quality and lineage metadata.

• AnalyticalDatasetdatasetId, partition, watermark, schemaVersion, qualityStatus, location

  Curated multi cloud data sync solution serving partition with freshness and quality state.

Data flow

1. 1. Register sources and contractsThe multi cloud data sync solution catalog records owners, schemas, compatibility rules, retention, lineage, and partitioning before data is accepted.
2. 2. Ingest with backpressureConnectors checkpoint multi cloud data sync solution source cursors, validate schema and deduplication keys, and slow producers when downstream capacity is exhausted.
3. 3. Process event time with checkpointsStream or batch engines compute multi cloud data sync solution transformations using watermarks, late-data policy, state checkpoints, and deterministic code versions.
4. 4. Publish quality-gated datasetsOnly multi cloud data sync solution outputs that pass completeness, freshness, validity, and privacy checks become visible to analytical consumers.
5. 5. Serve, replay, and reconcileConsumers read bounded partitions with freshness metadata while operators replay failed multi cloud data sync solution ranges and compare output checksums.

Deep dives and trade-offs

• Schema evolution and data qualityVersion multi cloud data sync solution contracts and make compatibility rules explicit for every producer and consumer. Quarantine malformed partitions instead of poisoning the whole dataset. Track row counts, null rates, duplicates, distribution changes, and policy violations by partition.
• Watermarks, late data, and exactly-once effectsUse source cursors and event-time watermarks for multi cloud data sync solution progress, not wall-clock assumptions. Make checkpoints, output keys, and sink commits retry-safe under at-least-once delivery. Document how late events revise windows, aggregates, or snapshots.
• Replay, lineage, and costKeep immutable multi cloud data sync solution raw evidence and code or schema versions so failed outputs can be reproduced. Separate hot serving storage from cold retention and cap replay concurrency. Measure freshness, backlog, compute cost, storage growth, and quality-gate failure rate.
• Streaming versus batchUse streaming for freshness-critical multi cloud data sync solution paths and batch for backfills, compaction, and expensive recomputation. Forcing every workload into streaming makes state, replay, and cost harder to operate.
• Raw retention versus curated-only storageRetain enough immutable raw evidence for replay, audit, and correction, then tier or expire it according to policy. Without raw evidence, a bad transformation can require an unreproducible emergency fix.
• Central warehouse versus domain-owned datasetsCentralize governance and discovery while letting domain owners own contracts and quality signals. A single team owning every transformation becomes a delivery bottleneck and hides data ownership.