← Back to the library
OperationsApplied · 8 min

Pods restart with OOMKilled and nothing in the app logs

“Pods restart every few hours or under load, kubectl shows OOMKilled with exit code 137, and the app logs simply stop with no error.”

ERROR CODES YOU MAY SEE

A code is a clue. Use its meaning and surrounding evidence to narrow the cause.

137Container exit codes · Kubernetes · platform

The container process was killed with SIGKILL (128 + 9). With reason OOMKilled, the kernel killed it for exceeding its cgroup memory limit.

Check lastState.terminated.reason in kubectl describe pod. OOMKilled means the memory limit; 137 without it can also be a kill after the termination grace period ran out.

Official reference for 137 (opens in new tab)Code definition reviewed

THE PRINCIPLE

The memory limit covers the whole process: heap, off-heap buffers, thread stacks, native libraries and page cache it cannot reclaim. Size the runtime's heap from the limit and leave headroom.

FIRST MOVES

  1. Confirm it is the limit: kubectl describe pod shows Last State Terminated, Reason OOMKilled, Exit Code 137. Node-level evictions show up as Evicted instead.
  2. Graph container_memory_working_set_bytes against the limit. A steady climb points to a leak; a sudden step points to one large request or batch.
  3. Check that the runtime sees the cgroup v2 limit. Node.js versions before 20.3 and older JDKs can size their heap from host memory; pin it with --max-old-space-size or -XX:MaxRAMPercentage.
  4. Leave about 25% of the limit for memory outside the heap. Set the memory request equal to the limit for critical pods so the scheduler does not overcommit the node.
  5. If it is a leak, take a heap snapshot under load in staging before raising the limit.

TOOLS: THEN → NOW

Fixed -Xmx or --max-old-space-size baked into the imageHeap derived from the container limit (-XX:MaxRAMPercentage, Node 20.3+ cgroup v2 detection), with an explicit cap where needed
Raising the limit after each crashWorking-set graphs and container_oom_events_total alerts, then a heap profile

PATTERN SNAPSHOT

resources:
  requests: { memory: "1Gi", cpu: "500m" }
  limits:   { memory: "1Gi" }        # request == limit: no memory overcommit
env:
  # Node: heap at ~75% of the limit; the rest is buffers, stacks, native code.
  - name: NODE_OPTIONS
    value: "--max-old-space-size=768"
  # JVM equivalent: JAVA_TOOL_OPTIONS="-XX:MaxRAMPercentage=75"

# Confirm from the pod:
#   cat /sys/fs/cgroup/memory.max     -> 1073741824
#   grep oom_kill /sys/fs/cgroup/memory.events

CLOSE THE AI. EXPLAIN THIS.

Why can a JVM or Node process get OOMKilled while its heap is well under the configured maximum?

HOW IT WORKS UNDERNEATH

SOURCES

Guide reviewed