Design a Log Anomaly Detection System — System Design Interview Practice

Interview prompt

Design a log-anomaly platform that normalizes heterogeneous logs, extracts templates, detects novel patterns, clusters related errors, correlates them with system events, and alerts operators with evidence.

• Keep raw logs immutable and derive normalized events, templates, embeddings, baselines, anomalies, correlations, and alert state.
• Use streaming template extraction and windowed baselines while controlling cardinality, retention cost, and false positives.
• Combine deterministic rules with statistical or ML detectors, attach representative log evidence, and support feedback loops.
• Explain late logs, schema drift, model cold start, alert deduplication, privacy, tenant isolation, and replay.

Requirements and scale assumptions

• Collect logs, parse formats, extract templates and fields, index normalized events, and calculate anomaly scores by service and window.
• Search by text or semantic similarity, show correlated deployments and metrics, group alerts, and provide representative evidence.
• Support detector configuration, feedback, suppression windows, retention, PII redaction, replay, and incident export.
• Process 1 million log lines per second with bounded agent overhead and detect high-confidence anomalies within one window.
• Scale search and detection by tenant, service, time, and template while isolating noisy services and high-cardinality fields.
• Make ingestion and alert delivery deduplicable; retain raw evidence so a detector can be replayed after a model change.
• Continue ingestion and rule-based detection when embeddings, model workers, or search indexes are degraded.
• Ingest 1 million lines per second from 10,000 services and retain searchable recent data plus archived raw logs.
• Partition by tenant, service, event time, and template; isolate bursty incidents and high-cardinality labels.
• Retain redacted raw events, parser versions, detector models, anomaly evidence, and alert decisions for replay.

• Peak scale: Support natural language search over logs — Capacity assumption that drives partitioning and backpressure.
• Latency target: Process 1M+ log lines per second — User-facing budget for the primary request or read path.
• Durable boundary: Committed before async — The source of truth is Parse and normalize heterogeneous log formats; Detect anomalous log patterns automatically.
• Async boundary: At-least-once workers — Keep Use Drain or LogMine for log template extraction, Implement word embeddings for semantic log analysis, Use autoencoders for unsupervised anomaly detection off the synchronous path.

Key entities

• DatasetVersiondatasetId, version, schemaHash, qualityStatus, lineage, createdAt

  Immutable log anomaly detection system input version used for reproducible training, evaluation, or replay.

• FeatureSnapshotentityId, featureSetVersion, eventTime, values, sourceWatermarks

  Point-in-time log anomaly detection system features with source watermarks so online and offline values can be compared.

• TrainingRunrunId, datasetVersion, codeVersion, metrics, artifactUri, status

  Audited log anomaly detection system run that records data, code, dependency, and evaluation lineage.

• ModelVersionmodelId, version, stage, schema, qualityGates, endpoint

  A promotable log anomaly detection system model version with rollout state, contract, and rollback metadata.

Data flow

1. 1. Register and validate training dataThe log anomaly detection system gateway records an immutable dataset version, schema, lineage, quality status, and privacy disposition.
2. 2. Build point-in-time featuresFeature workers join log anomaly detection system inputs using event-time watermarks, prevent leakage, and publish the same feature contract for training and serving.
3. 3. Train and evaluate asynchronouslyThe orchestrator schedules log anomaly detection system runs with checkpointed artifacts, reproducible environments, and metrics tied to the exact input versions.
4. 4. Gate and serve a model versionA registry compares log anomaly detection system quality, bias, safety, and compatibility gates before canary or production rollout with an immediate rollback pointer.
5. 5. Monitor drift and learn from feedbackOnline inference records latency, errors, drift, and delayed labels so log anomaly detection system retraining is evidence-driven rather than triggered by guesswork.

Deep dives and trade-offs

• Reproducibility and leakage preventionPin log anomaly detection system data, feature, code, dependency, and model versions for every run. Use point-in-time joins and quarantine failed quality or privacy checks before training. Keep raw inputs and artifacts immutable so a result can be replayed after a dependency changes.
• Safe promotion and serving contractsSeparate log anomaly detection system model registration from deployment and require signed artifacts plus schema compatibility. Use shadow traffic, canaries, rollback pointers, and per-version latency/error budgets. Return model version and feature freshness so clients can explain or reproduce a prediction.
• Drift, feedback, and costMeasure feature drift, prediction drift, label delay, and segment-level quality for log anomaly detection system rather than only aggregate accuracy. Sample expensive inference and cap retraining concurrency with an explicit GPU or compute budget. Keep human corrections and delayed labels linked to the original prediction and model version.
• Batch versus online featuresPrefer a shared feature contract with batch backfills and a low-latency online serving path for decisions that need freshness. Two independently defined transformations create training-serving skew and hard-to-debug regressions.
• Synchronous versus asynchronous inferenceKeep interactive log anomaly detection system inference synchronous within a strict budget and queue large or expensive jobs. A request path that waits for model loading, enrichment, or retraining turns downstream slowness into an outage.
• Global model versus segment modelsStart with one versioned model and add segment-specific models only when quality or policy evidence justifies the operational cost. Many simultaneously active versions multiply monitoring, rollback, and data-lineage burden.