← Back to the library
OperationsDeep dive · 9 min

Latency spikes while CPU usage looks low

“p99 latency jumps in steps of tens of milliseconds, but the dashboard shows the pod at 30-40% of its CPU limit and nothing is saturated.”

THE PRINCIPLE

Use CPU requests to guarantee capacity and treat CPU limits as a deliberate choice. Size the runtime's thread pools to the limit, not to the node's core count.

FIRST MOVES

  1. Graph rate(container_cpu_cfs_throttled_periods_total) / rate(container_cpu_cfs_periods_total) per container. A sustained ratio above a few percent on a latency-sensitive service is the cause, not a coincidence.
  2. Compare thread counts to the limit: a 2-CPU limit on a 64-core node lets a runtime start 64 GC or worker threads that burn the quota in a few milliseconds.
  3. Set the runtime to match the limit: Go 1.25+ does this by default (older Go: automaxprocs). For the JVM, check -XX:ActiveProcessorCount and the GC thread counts. For Node, size worker_threads and cluster pools to the limit instead of os.cpus().length.
  4. Raise the limit or remove it for latency-critical services (without a limit, runtimes size themselves to the node, so set thread counts explicitly), and keep requests accurate so the scheduler and HPA still work. Keep limits where noisy neighbours or cost caps matter more.
  5. Look at cpu.stat inside the container (nr_throttled, throttled_usec) to confirm, rather than relying on usage graphs alone.

TOOLS: THEN → NOW

CPU usage averaged over 1-5 minutesThrottled-period ratio from cAdvisor/kubelet metrics next to p99 latency
Thread pools sized to the node's coresRuntimes that read the cgroup limit (Go 1.25 GOMAXPROCS, container-aware JDKs) or explicit settings

PATTERN SNAPSHOT

# Fraction of 100ms periods in which the container was throttled.
sum by (namespace, pod, container) (
  rate(container_cpu_cfs_throttled_periods_total{container!=""}[5m])
) /
sum by (namespace, pod, container) (
  rate(container_cpu_cfs_periods_total{container!=""}[5m])
) > 0.05

# Inside the pod (cgroup v2):
#   cat /sys/fs/cgroup/cpu.max    -> "200000 100000" = 2 CPUs per 100ms
#   cat /sys/fs/cgroup/cpu.stat   -> nr_periods, nr_throttled, throttled_usec

CLOSE THE AI. EXPLAIN THIS.

A pod with a 1-CPU limit runs 8 busy threads for 20 ms. What happens to every request in flight for the rest of that 100 ms period?

HOW IT WORKS UNDERNEATH

SOURCES

Guide reviewed