Your Kubernetes Pod Restarted. Now What?

A restart is Kubernetes doing its job. The container stopped, and the kubelet started it again according to the pod’s restart policy. The question is never “why did Kubernetes restart it?” It is “why did the container stop?”

This episode walks through that question in order, from the cheapest evidence to the most detailed, using an example service called checkout whose containers keep restarting.

The Signal

The first signal is the RESTARTS column:

kubectl get pods -n shop

NAME                        READY   STATUS    RESTARTS      AGE
checkout-6c8d9b7f5d-4xk2p   1/1     Running   4 (12m ago)   3h
checkout-6c8d9b7f5d-9wq7m   1/1     Running   3 (31m ago)   3h
checkout-6c8d9b7f5d-tj5rn   1/1     Running   4 (8m ago)    3h

Three replicas, all Running, all restarting every so often. A dashboard of pod status would show green most of the time, which is exactly why restart counts deserve their own alert.

The Problem

A restart count tells you that containers stopped. It does not tell you why. Common reasons look identical in that column:

What happened	Where you see it	Typical exit code
Container exceeded its memory limit	lastState.terminated.reason: OOMKilled	137
Application exited with an error	reason: Error and the app’s own logs	Whatever the app returned, often 1
Liveness probe failed repeatedly	Unhealthy and Killing events on the pod	Often 137 or 143, depending on how the process handles SIGTERM
Process exited successfully but should not have	reason: Completed	0

Exit code 137 means the process received SIGKILL (128 + 9). The out-of-memory killer is the most common source, but it is not the only one: a container that ignores SIGTERM during a liveness restart is also killed with SIGKILL when its grace period ends. Read the reason, not just the code.

Node-pressure eviction is different again. An evicted pod is terminated and replaced by its controller with a new pod, so it shows up as a pod with status Evicted (or a replacement pod with a new name), not as a higher restart count.

Beyond the Dashboard

Most dashboards show the current state: running pods, current memory, current CPU. The evidence you need belongs to the container instance that already died. Kubernetes keeps a little of it, briefly:

• the last termination state of each container, in the pod status;
• events, which by default are retained for about an hour;
• the previous container’s logs, until it restarts again.

Collect them before they rotate away.

The Investigation

1. Read the termination reason

kubectl describe pod checkout-6c8d9b7f5d-tj5rn -n shop

    State:          Running
      Started:      ...
    Last State:     Terminated
      Reason:       OOMKilled
      Exit Code:    137
      Started:      ...
      Finished:     ...
    Restart Count:  4
    Limits:
      memory:  256Mi
    Requests:
      cpu:     100m
      memory:  256Mi

For scripts and runbooks, the same data is available as JSON:

kubectl get pod checkout-6c8d9b7f5d-tj5rn -n shop \
  -o jsonpath='{range .status.containerStatuses[*]}{.name}{"\t"}{.lastState.terminated.reason}{"\t"}{.lastState.terminated.exitCode}{"\n"}{end}'

Here the answer is unambiguous: OOMKilled. The container used more memory than its 256Mi limit, and the kernel killed it.

2. Check the events

kubectl get events -n shop \
  --field-selector involvedObject.name=checkout-6c8d9b7f5d-tj5rn \
  --sort-by=.lastTimestamp

Events add context, but they do not tell the whole story. An OOM kill does not produce a dedicated pod event; you will mostly see the container being started again, and BackOff if it is crash-looping. What events are good at is ruling in other causes: Unhealthy and Killing events from a liveness probe mean you are looking at a probe problem, not memory.

3. Read the logs of the container that died

kubectl logs checkout-6c8d9b7f5d-tj5rn -n shop --previous

--previous returns the logs of the last terminated container. With an OOM kill, the log usually just stops: the process gets no chance to write a final message. That absence is itself a clue. If logs are shipped to a central store, filter by the pod name and the time window before the restart instead.

4. Look at memory over time

The termination reason says what happened. Metrics show how. With cAdvisor and kube-state-metrics in Prometheus, compare working set memory against the limit:

max by (pod) (
  container_memory_working_set_bytes{namespace="shop", container="checkout"}
)
/
max by (pod) (
  kube_pod_container_resource_limits{namespace="shop", container="checkout", resource="memory"}
)

The shape matters more than any single value:

• A steady climb that resets at every restart suggests a leak or an unbounded cache.
• Flat, then a sudden spike suggests a specific request or job that allocates a lot at once.
• Consistently close to the limit suggests the limit is simply too small for normal load.

In our example, memory climbs steadily after each restart until it reaches the limit: the first pattern.

To alert on restarts and their reason, kube-state-metrics exposes both:

increase(kube_pod_container_status_restarts_total{namespace="shop"}[1h]) > 0

kube_pod_container_status_last_terminated_reason{namespace="shop", reason="OOMKilled"} == 1
and on (namespace, pod, container)
increase(kube_pod_container_status_restarts_total{namespace="shop"}[10m]) > 0

5. Find what users saw

Restarts are not free for users. Requests in flight on the killed container fail, and new requests may be routed to it before it is ready again. If the service is instrumented with OpenTelemetry and its spans carry the k8s.pod.name resource attribute, you can:

• filter traces by the pod name and the minute before each restart, and look for spans that never completed, or errors seen by the caller (connection resets, 502s or 503s at the ingress);
• check which routes those requests were on. In our example, the requests before each kill cluster on an export endpoint that builds large responses in memory.

A continuous profiler, if you have one, is the next step: an allocation profile of the checkout container points directly at the code holding the memory.

The Engineering Decision

There are several options, and they are not mutually exclusive:

1. Fix the allocation. In our example, that means streaming the export response instead of building it in memory. This is the real fix, and usually the slowest one.
2. Raise the memory limit. This buys time, and is right when the limit was set without data. It is wrong when memory grows without bound, because it only delays the next kill.
3. Tune the runtime to the limit. Modern JVMs read the container limit and default to a heap of 25% of it, so -XX:MaxRAMPercentage is usually about giving the heap more room, not less. Go’s garbage collector ignores the limit until you set GOMEMLIMIT, which makes it collect harder before the kernel steps in. Check what your runtime actually does.
4. Alert before the limit, not after the kill. Alert when working set stays above a threshold of the limit, or when its trend predicts reaching the limit within hours. The alert that saved the weekend is this practice in story form.
5. Make the restart itself cheaper. An OOM kill is immediate: no preStop hook, no grace period. What helps is enough replicas that one restarting container is a small share of traffic, clients that retry idempotent requests, and a readiness probe so the restarted container only gets traffic once it can serve. (For planned restarts, such as rollouts and node drains, a preStop hook, a sensible terminationGracePeriodSeconds, and a PodDisruptionBudget do matter.)

The Outcome

This is a hypothetical example, so there is no measured result to report. To validate a fix like this, reproduce the problem first (a load test that exercises the suspect endpoint is usually enough), apply the change, and run the same test again. Then watch your own traffic for at least one full cycle: restart counts, the memory-to-limit ratio, and caller-side errors should all stay flat.

Lessons from the Field

• A restart is a symptom. Read lastState.terminated.reason first, and the exit code second.
• Exit code 137 means SIGKILL, which is usually, but not always, the OOM killer.
• Evidence about a dead container is short-lived. Capture events and previous logs early, and ship both to central storage so they outlive the pod.
• The shape of memory over time tells you whether you have a leak, a spike, or a limit that is too small.
• Connect restarts to user impact with traces that carry Kubernetes resource attributes.
• Turn the investigation into a runbook. The runbook that worked shows why.