Design an AI-Powered Cloud Cost Optimization Platform — System Design Interview Practice

Interview prompt

Design a multi-cloud FinOps platform that normalizes billing and usage, allocates costs, detects waste/anomalies, recommends rightsizing or commitments, and forecasts spend with explainable evidence.

• Define normalized billing line items, usage/resource identity, ownership/tag policy, allocation rules, commitments, recommendations, and realized savings.
• Reconcile provider invoices with usage, handle credits/taxes/currency, delayed data, shared resources, and changing prices without hiding uncertainty.
• Separate ingestion and normalization from dashboards, forecasting, recommendation computation, approval workflows, and provider-side changes.
• Explain recommendation safety, stale billing, privacy/tenant isolation, auditability, feedback, observability, and degraded reporting.

Requirements and scale assumptions

• Ingest multi-cloud billing/usage data, normalize resources and currencies, allocate shared spend, and expose daily/monthly cost views.
• Detect anomalies and idle resources, forecast spend, recommend rightsizing/commitments, model scenarios, and track approved savings.
• Support budgets/alerts, showback/chargeback, recommendation approval, provider action rollback where possible, corrections, and audit exports.
• Publish cost data within 24 hours of provider availability and show recommendation confidence, assumptions, and expected savings.
• Process billions of billing line items across providers without a single hot key or unbounded synchronous work.
• Do not lose committed state; make retries and duplicate events safe.
• Degrade safely when downstream workers, caches, or external dependencies fail.
• 1B billing line items/month across 10k accounts and 3 cloud providers
• Partition raw and normalized data by tenant, provider, account, and billing period; shard large accounts and hot shared services.
• Keep serving state bounded; retain raw events or durable records for replay and auditing.

• Peak scale: 1B line items/month; 10k accounts — Capacity assumption that drives partitioning and backpressure.
• Latency target: dashboard p95 < 2s; billing freshness < 24h — Interactive spend and recommendation reads are served from projections; every result carries its source and projection watermark.
• Durable boundary: Committed before async — Provider billing records and approved allocation rules are authoritative; forecasts and recommendations are derived.
• Async boundary: At-least-once batch and stream workers — Keep billing ingestion, allocation, forecasting, anomaly detection, rightsizing, commitment analysis, and index refresh off the synchronous path.

Key entities

• BillingLineItemlineItemId, tenantId, provider, accountId, service, resourceId, usageStart, usageEnd, netCost, currency, sourceVersion

  Immutable provider evidence kept with source identifiers, credits, taxes, rate context, and collection watermark so invoice totals can be reconciled.

• ResourceOwnershiptenantId, provider, resourceId, ownerId, team, businessUnit, tags, region, effectiveFrom, effectiveTo

  Versioned mapping used for showback and chargeback; unknown ownership is surfaced as unallocated.

• CostSnapshotsnapshotId, tenantId, accountId, period, grouping, allocatedCost, unallocatedCost, allocationVersion, sourceWatermark

  Serving aggregate keyed by account and period with allocation coverage and provider watermark.

• OptimizationRecommendationrecommendationId, tenantId, resourceId, recommendationType, evidenceRefs, estimatedMonthlySavings, confidence, modelVersion, policyVersion, status, approvalExpiresAt, actionId

  Versioned explainable candidate for idle cleanup, rightsizing, or commitment purchase.

Data flow

1. 1. Connect cloud accounts and collect evidenceProvider adapters use scoped credentials and provider cursors to land billing exports, usage, rates, credits, and resource inventory while respecting provider quotas.
2. 2. Preserve raw billing and reconcile invoicesThe billing lake keeps immutable provider-native partitions; reconciliation compares invoice totals, credits, taxes, and exchange-rate inputs before a period is marked complete.
3. 3. Normalize and allocate costWorkers convert line items to a canonical model, apply amortization and currency rules, join effective ownership, and publish explicit unallocated spend for missing metadata.
4. 4. Detect waste and generate explainable recommendationsFeature workers build utilization and spend windows; forecasting, anomaly, idle-resource, rightsizing, and commitment models produce evidence-backed candidates.
5. 5. Review, execute, and measure savingsPolicy rules and user approval gate mutations; the executor performs guarded provider actions and later compares billing and usage with the baseline.

Deep dives and trade-offs

• Billing correctness and late dataFor the multi-cloud FinOps platform, provider evidence, allocation coverage, model assumptions, and realized savings must remain distinguishable. Design for the failure case where late billing, missing ownership, stale utilization, or an unsafe approval must not create a false saving or outage; keep retries, versions, and repair state explicit. Expose freshness, version, lineage, or audit metadata so operators and clients can distinguish current, pending, and degraded state.
• Multi-cloud allocation and feature historyFor the multi-cloud FinOps platform, provider evidence, allocation coverage, model assumptions, and realized savings must remain distinguishable. Keep this concern off unrelated request paths and partition it by the multi-cloud FinOps platform access key. Expose freshness, version, lineage, or audit metadata so operators and clients can distinguish current, pending, and degraded state.
• Recommendation safety and feedbackFor the multi-cloud FinOps platform, provider evidence, allocation coverage, model assumptions, and realized savings must remain distinguishable. Keep this concern off unrelated request paths and partition it by the multi-cloud FinOps platform access key. Expose freshness, version, lineage, or audit metadata so operators and clients can distinguish current, pending, and degraded state.
• Raw lake plus serving index versus one databaseKeep immutable provider evidence and canonical aggregates in durable analytical storage, with a separate tenant-filtered index for dashboards. Two representations require watermarks and reconciliation, but a single database cannot serve billion-line-item history cheaply.
• Scheduled batch versus continuous analysisUse provider-aligned batch ingestion for invoice-grade cost and a smaller streaming path for budget anomalies and action results. Continuous polling hits provider quotas and creates false precision when billing data itself is delayed.
• Automatic remediation versus human approvalAutomate low-risk detection and notification, but require policy, dry run, and scoped approval for production mutations or commitments. Stale utilization, tags, or provider prices can turn automation into an outage or financial commitment.