Logging & Monitoring

InfrastructureSenior

The Question

What are best practices for logging and monitoring in production?

What a Strong Answer Covers

• ELK components
• logging vs monitoring vs APM distinction.

Senior-Level Answer

Production observability rests on three pillars—logs, metrics, and traces—and effective logging and monitoring practices make the difference between a 5-minute and 5-hour incident resolution.

**Structured logging**: Emit logs as JSON objects, not free-text strings. Include consistent fields: `timestamp` (ISO 8601 UTC), `level`, `service`, `trace_id`, `request_id`, `user_id` (where applicable), `message`, and context-specific fields. Structured logs are machine-parseable, enabling fast filtering and aggregation in log platforms (Datadog, Splunk, CloudWatch Insights). Free-text logs require regex parsing, which breaks on message changes.

**Log levels with discipline**: DEBUG for development noise (never in production by default), INFO for significant state transitions (request received, job completed), WARN for recoverable unexpected conditions, ERROR for failures requiring attention, CRITICAL/FATAL for system-level failures. Tune log level per environment—DEBUG in staging, INFO in production. Never log sensitive data (passwords, PII, tokens) at any level.

**Correlation IDs**: Propagate a `trace_id` or `request_id` through every service call for a given user action. Log it at every layer. This enables reconstructing the full call path across microservices during an incident. Standards like W3C `traceparent` and OpenTelemetry make this interoperable.

**Metrics**: Logs answer "what happened"; metrics answer "how often / how fast". Instrument with the RED method (Rate, Errors, Duration) per service and the USE method (Utilization, Saturation, Errors) per resource. Export via Prometheus, StatsD, or OTLP. Dashboards in Grafana or Datadog surface trends; alerts fire on thresholds.

**Distributed tracing**: Tools like Jaeger, Zipkin, or Datadog APM stitch together spans across service boundaries. Essential for identifying latency contributions in microservice architectures.

**Log retention and cost**: Production logs have cost at scale. Apply sampling for high-volume debug logs (1% of INFO, 100% of ERROR). Set retention policies (30 days hot, archive to S3 for compliance).

**Alerting on symptoms, not causes**: Alert on user-visible impact (error rate > 1%, p99 latency > 500ms) rather than internal metrics (CPU > 80%). Symptom-based alerts reduce noise and map directly to SLOs.

What Separates a 2/3 from a 3/3

Common Mistakes

• Logging sensitive data (tokens, PII) without redaction.
• Using free-text log messages that resist machine parsing.
• Not propagating a trace/request ID across service boundaries.
• Alerting on resource metrics (CPU, memory) instead of user-visible symptoms (error rate, latency).

Follow-Up Questions

• How do you implement log sampling without losing visibility into errors? — Sample low-severity high-volume logs (DEBUG/INFO) at a fraction; always log WARN/ERROR at 100%. Reservoir sampling for INFO ensures representative coverage.
• What is the difference between a log, a metric, and a trace? — Log: timestamped event record (what happened). Metric: aggregated numeric measurement over time (how often/fast). Trace: end-to-end path of a single request across services.
• How would you debug a latency spike using all three observability pillars? — Metrics alert on p99 spike → trace shows which service added latency → logs in that service show the slow query or external call that caused it.
• How do you prevent logs from becoming a PII compliance liability? — Scrub/mask PII at the log emitter before writing. Use structured fields so redaction is deterministic. Audit log pipeline configuration regularly.

Related Questions

• K8s — Pod vs Deployment
• K8s — Liveness vs Readiness
• K8s — ConfigMap vs Secret
• Helm Chart
• Docker — Container vs VM