Design a Secure Multi-Tier Web Application — System Design Interview Practice

Interview prompt

Design a secure multi-tier web application with isolated network zones, protected ingress, private data access, elastic capacity, strong identity controls, and auditable operations.

• Define public ingress, web, application, and data tiers with least-privilege flows, explicit trust boundaries, and no direct database exposure.
• Cover TLS lifecycle, WAF/rate limits, identity propagation, secrets, patching, supply-chain controls, and administrative access.
• Explain autoscaling, connection pooling, backups, multi-zone failover, idempotent mutations, and safe handling of dependency failures.
• Include threat modeling, audit evidence, detection/response, observability, and a degraded mode.

Requirements and scale assumptions

• Serve public requests through protected ingress to stateless application services and private data stores, with authenticated user operations.
• Support role-based administration, secrets/certificate rotation, audit logs, health checks, scaling policies, and controlled deployments.
• Provide backups and restore tests, zone failover, rate-limit responses, replay-safe writes, and incident isolation when a tier is compromised.
• Meet p95 API latency under 200ms while maintaining zero direct internet access to the database tier.
• Scale web and application tiers independently across multiple zones without a single hot key or unbounded synchronous work.
• Do not lose committed state; make retries and duplicate events safe.
• Degrade safely when downstream workers, caches, or external dependencies fail.
• Three availability zones; 10k requests/second peak
• Partition by the primary tenant, user, item, or geographic key and isolate hot partitions.
• Keep serving state bounded; retain raw events or durable records for replay and auditing.

• Peak scale: 10k requests/s across 3 zones — Capacity assumption that drives partitioning and backpressure.
• Latency target: API p95 < 200ms; DB exposure = 0 — User-facing budget for the primary request or read path.
• Durable boundary: Committed before async — The primary database is authoritative; caches, sessions, and read replicas are constrained projections.
• Async boundary: At-least-once workers — Keep VPC with public and private subnets, Application Load Balancer for traffic, Auto Scaling Group for web tier off the synchronous path.

Key entities

• AccessSessionsessionId, principalId, roles, credentialVersion, expiresAt

  Short-lived authenticated session for secure multi tier web application with revocation and rotation metadata.

• SecureWebResourcerecordId, ownerId, status, version, createdAt, updatedAt

  Authoritative private application state protected by network-zone and role policy.

• AuditEventeventId, principalId, action, resourceId, policyVersion, occurredAt

  Tamper-evident access, policy, and deployment evidence for secure multi tier web application.

• MigrationCheckpointmigrationId, partition, sourceVersion, targetVersion, checkpoint, status

  Replayable secure multi tier web application migration or recovery progress.

Data flow

1. 1. Protect ingress and authenticateThe secure multi tier web application edge validates transport, identity, roles, quotas, and network-zone policy before routing the request.
2. 2. Authorize and commit private stateThe application enforces resource policy and conditionally commits secure multi tier web application state before returning a durable version.
3. 3. Publish audit and domain eventsAn outbox emits secure multi tier web application changes, access evidence, and migration work only after the private store commit.
4. 4. Run migrations and side effects asynchronouslyPartitioned workers process secure multi tier web application exports, notifications, migrations, backups, and repairs with checkpoints and bounded retries.
5. 5. Fail over and restore safelyHealth checks, backups, restore tests, replay, and audit evidence recover secure multi tier web application after a tier or zone failure.

Deep dives and trade-offs

• Network isolation and identityPlace secure multi tier web application data stores in private zones and expose only narrow service-to-service paths. Use short-lived credentials, role checks, secret or certificate rotation, and deny-by-default policies. Log policy decisions without putting tokens or sensitive payloads into traces.
• Safe migrations and concurrencyVersion secure multi tier web application records and use expand-migrate-contract for schema changes. Checkpoint large migrations by partition and make retries idempotent. Reject stale writes rather than allowing a slower tier to overwrite newer state.
• Backups, restore, and degraded readsTest secure multi tier web application restore procedures, not just backup creation, with measured recovery objectives. Keep caches and projections rebuildable from private authoritative state. Expose degraded or read-only mode when a write dependency or zone is unavailable.
• Global active-active versus regional ownershipKeep secure multi tier web application writes regionally owned or partitioned and replicate derived reads globally. Cross-region active-active writes increase conflict, identity, and recovery complexity.
• Synchronous versus asynchronous side effectsCommit the protected business state synchronously and queue audit exports, notifications, and migrations. Making every side effect synchronous widens the failure domain and leaks provider latency into user requests.
• Cache availability versus data freshnessCache only safe reads with TTL and invalidation metadata, and fall back to the private store for critical decisions. A stale cache must never bypass authorization or become the only copy of data.