Build a Climate Data Analysis Platform — System Design Interview Practice

Interview prompt

Design a scientific climate-data platform that ingests satellite imagery, runs geospatial analysis and climate models, and serves reproducible visualization dashboards over decades of observations.

• Separate immutable satellite scenes and derived raster products from catalog metadata, model runs, and dashboard caches.
• Partition imagery by spatial tile, acquisition time, and collection so petabyte-scale processing can run in parallel.
• Make geospatial jobs asynchronous, reproducible, and resumable while keeping small catalog and visualization queries fast.
• Explain coordinate reference systems, missing scenes, quality masks, provenance, access control, and regional recovery.

Requirements and scale assumptions

• Ingest satellite scenes and station observations, validate metadata, and create searchable collections.
• Run tiled raster transforms, temporal aggregations, and climate-model jobs with progress and reproducible outputs.
• Display map layers, time series, provenance, quality flags, and exportable results for authorized users.
• Keep catalog and tile metadata reads below 300 ms p95; expose long-running model jobs asynchronously.
• Process petabyte-scale archives without a single hot spatial tile or unbounded synchronous work.
• Do not lose committed scene metadata or provenance; make tiled retries and duplicate ingestion safe.
• Degrade safely when model workers or external imagery providers fail and preserve the last valid published layer.
• Petabyte-scale archive across decades of satellite scenes and derived products.
• Partition by spatial tile, acquisition date, sensor, and collection; split popular tiles by time or resolution.
• Keep raw scenes immutable, retain manifests and provenance, and cache only bounded metadata and rendered tiles.

• Archive scale: Petabytes — Raw scenes and derived products drive tile partitioning, lifecycle tiers, and backpressure.
• Tile query latency: p95 <=300ms — Catalog and cached tile metadata should remain interactive while analysis runs in jobs.
• Durable boundary: Committed before async — The source of truth is immutable scene metadata, object manifests, and versioned model outputs.
• Async boundary: At-least-once workers — Keep tiled geospatial transforms, model execution, and warehouse loads off the synchronous path.

Key entities

• SourcePartitionsourceId, partitionId, cursor, schemaVersion, watermark, status

  Replayable climate data analysis platform source evidence and ingestion cursor.

• SchemaVersiondatasetId, version, compatibility, owner, effectiveAt, status

  Governed climate data analysis platform contract used to validate producers and consumers.

• ProcessingRunrunId, inputWatermark, checkpoint, qualityStatus, codeVersion, status

  Checkpointed climate data analysis platform processing attempt with quality and lineage metadata.

• AnalyticalDatasetdatasetId, partition, watermark, schemaVersion, qualityStatus, location

  Curated climate data analysis platform serving partition with freshness and quality state.

Data flow

1. 1. Register sources and contractsThe climate data analysis platform catalog records owners, schemas, compatibility rules, retention, lineage, and partitioning before data is accepted.
2. 2. Ingest with backpressureConnectors checkpoint climate data analysis platform source cursors, validate schema and deduplication keys, and slow producers when downstream capacity is exhausted.
3. 3. Process event time with checkpointsStream or batch engines compute climate data analysis platform transformations using watermarks, late-data policy, state checkpoints, and deterministic code versions.
4. 4. Publish quality-gated datasetsOnly climate data analysis platform outputs that pass completeness, freshness, validity, and privacy checks become visible to analytical consumers.
5. 5. Serve, replay, and reconcileConsumers read bounded partitions with freshness metadata while operators replay failed climate data analysis platform ranges and compare output checksums.

Deep dives and trade-offs

• Schema evolution and data qualityVersion climate data analysis platform contracts and make compatibility rules explicit for every producer and consumer. Quarantine malformed partitions instead of poisoning the whole dataset. Track row counts, null rates, duplicates, distribution changes, and policy violations by partition.
• Watermarks, late data, and exactly-once effectsUse source cursors and event-time watermarks for climate data analysis platform progress, not wall-clock assumptions. Make checkpoints, output keys, and sink commits retry-safe under at-least-once delivery. Document how late events revise windows, aggregates, or snapshots.
• Replay, lineage, and costKeep immutable climate data analysis platform raw evidence and code or schema versions so failed outputs can be reproduced. Separate hot serving storage from cold retention and cap replay concurrency. Measure freshness, backlog, compute cost, storage growth, and quality-gate failure rate.
• Streaming versus batchUse streaming for freshness-critical climate data analysis platform paths and batch for backfills, compaction, and expensive recomputation. Forcing every workload into streaming makes state, replay, and cost harder to operate.
• Raw retention versus curated-only storageRetain enough immutable raw evidence for replay, audit, and correction, then tier or expire it according to policy. Without raw evidence, a bad transformation can require an unreproducible emergency fix.
• Central warehouse versus domain-owned datasetsCentralize governance and discovery while letting domain owners own contracts and quality signals. A single team owning every transformation becomes a delivery bottleneck and hides data ownership.