An alert says p99 latency doubled. CPU looks normal. Error rate is low. The fastest way to make the incident longer is to start changing things based on whichever dashboard is already open.

Latency is a budget distributed across queueing, application work, storage, networks, and dependencies. The incident task is to find which part changed and whether the change affects all traffic or one slice.

Stabilize before explaining

First establish user impact and protect the system:

  • confirm which routes, regions, tenants, and versions are affected;
  • compare p50, p95, and p99 rather than one average;
  • pause a rollout if timing and scope align;
  • reduce optional work, shed load, or apply a safe capacity increase;
  • preserve traces, logs, deploy identifiers, and database evidence.

Mitigation and root-cause analysis can run in parallel, but every mitigation should be reversible and recorded.

Build a latency decomposition

For one slow request, the total can be modeled as:

request latency =
    queue wait
  + application compute
  + database and cache calls
  + downstream service calls
  + serialization and network transfer

A distributed trace should make this visible, but tracing can lie by omission if sampling drops slow requests or a library is not instrumented. Reconcile traces with service-level request timers and dependency metrics.

Segment before aggregating

An aggregate p99 may be driven by one low-volume route. Break the signal down by:

  • route or operation;
  • status code;
  • region and availability zone;
  • application version;
  • tenant or workload class;
  • cache hit versus miss;
  • database query fingerprint.

Avoid unbounded labels such as raw user IDs in metrics. Use logs or trace attributes for high-cardinality investigation.

An average can remain flat while the tail collapses. Imagine 99 requests complete in 100 ms and one completes in 10 seconds. The mean is about 199 ms; p99, depending on the quantile convention and sample size, sits near the slow request. If the single slow request becomes five, the mean moves modestly while p95 and p99 expose a different user experience.

Do not compare percentiles by averaging pre-aggregated percentiles across hosts. Quantiles are not composable that way. Aggregate histograms with consistent buckets or calculate the quantile from the combined distribution.

Look for saturation and queueing

CPU below 100% does not prove spare capacity. Check:

  • event-loop lag or thread-pool queue depth;
  • database connection-pool wait time;
  • worker and message-queue backlog;
  • container CPU throttling;
  • memory pressure and garbage-collection pauses;
  • downstream concurrency limits.

Queueing rises sharply as utilization approaches capacity. A service can have moderate average CPU while one shared pool is exhausted.

Little’s Law gives a useful consistency check:

concurrency = throughput × time_in_system

At 500 requests per second and 200 ms end-to-end latency, expect roughly 100 concurrent requests. If observed concurrency is 400 at the same throughput, either the latency measurement misses waiting time, a retry/duplicate path is inflating work, or the system is holding requests somewhere the dashboard does not expose.

This is not a root-cause formula. It is a way to catch an incomplete model.

Interrogate the data path

For databases, compare query fingerprints before and after the spike. Look for changed plans, missing indexes, lock waits, connection acquisition, rows scanned, and result size.

For caches, separate hit latency from miss amplification. A small drop in hit rate can multiply database traffic. If many requests rebuild the same expired entry, the real issue is a cache stampede, not cache speed.

Useful protections include request coalescing, stale-while-revalidate, randomized expiry, and bounded concurrency on cache rebuilds.

Correlate with change, but do not stop there

Deploys, feature flags, traffic mix, data growth, certificate rotation, dependency incidents, and scheduled jobs are all change. A code deploy is only one possibility.

Create a timeline with:

  1. first observed user impact;
  2. metric deviation;
  3. deploy and configuration events;
  4. dependency or infrastructure changes;
  5. mitigation and measured response.

The timeline prevents a compelling theory from silently becoming a fact.

Run the investigation as competing hypotheses

For a spike isolated to cache misses, I would keep at least three hypotheses alive:

  1. database service time increased;
  2. the cache hit rate fell, amplifying otherwise normal database work;
  3. requests are queued before the database call because the connection pool is saturated.

Each predicts different evidence. Hypothesis one predicts slower database spans and query execution. Hypothesis two predicts increased miss volume followed by database traffic, possibly without slower individual queries. Hypothesis three predicts connection-acquisition time rising while execution time stays stable.

Write the prediction before opening the next dashboard. This avoids moving the theory every time new evidence appears.

Instrument the missing boundary

A single db.duration timer is insufficient when it combines pool wait, network time, execution, and row decoding. Instrument:

db.pool_wait_ms
db.execute_ms
db.rows_returned
db.query_fingerprint
cache.outcome = hit | miss | stale
queue.wait_ms
dependency.name
dependency.status

Keep query fingerprints normalized; raw SQL can contain sensitive values and creates high cardinality. For traces, record identifiers required to correlate a request while applying a deliberate sampling and retention policy.

Turn the cause into a guardrail

“A query was slow” is not a complete root cause. Ask why the query reached production, why its cost increased, why protection failed, and why detection took as long as it did.

A useful follow-up might include:

  • a query-plan regression test;
  • a connection-pool wait alert;
  • a per-route latency objective;
  • a load test with realistic data distribution;
  • a circuit breaker or concurrency bound;
  • a rollout check comparing new and old versions.
The goal of an incident review is not a more detailed story. It is a smaller class of incidents that can happen again.

Production debugging becomes faster when observability mirrors the system’s actual resource boundaries. Without that model, teams collect more telemetry and still guess.

References