How to Fix CrashLoopBackOff in Kubernetes: A Step-by-Step Playbook for the 3 AM Page

What CrashLoopBackOff actually means (and what it doesn’t)

CrashLoopBackOff is not an error. It’s a waiting state. Your container started, exited, got restarted, exited again, and the kubelet is now pausing between attempts so it doesn’t hammer your node.

Per the Kubernetes Pod lifecycle docs, the kubelet restarts the container with an exponential back-off delay of 10s, 20s, 40s and so on, capped at 300 seconds (5 minutes), and resets the timer once the container has run for 10 minutes without problems. That’s why a loop feels slow: by restart six you’re waiting five minutes for each new attempt. The same page documents alpha feature gates (ReduceDefaultCrashLoopBackOffDecay and KubeletCrashLoopBackOffMax) that shorten the delay, but the defaults are what most clusters run.

Whether it restarts at all depends on restartPolicy: Always (the default, and the only option for Deployments), OnFailure or Never. Before you debug, make sure you’re actually looking at a crash loop and not one of its lookalikes:

The key distinction: CrashLoopBackOff means your code ran. The image pulled, the config resolved, the process started. So the answer is almost always inside the container’s own output.

How to fix CrashLoopBackOff in Kubernetes, step by step

Same seven steps, every time. Swap in your own namespace and Pod names. The goal of steps 1 to 5 is evidence; only step 7 changes anything.

1. 1

   Confirm the loop and find the container

   List the Pod and note the RESTARTS count and which container is crashing. Multi-container Pods hide the culprit.

   kubectl get pods -n shop
   kubectl get pod checkout-7d9f8b6c4-x2k9p -n shop \
     -o jsonpath='{range .status.containerStatuses[*]}{.name}{"\t"}{.restartCount}{"\n"}{end}'

2. 2

   Read the last state: reason and exit code

   kubectl describe shows Last State: Terminated with a Reason and an Exit Code. This one line decides which branch of the playbook you take.

   kubectl describe pod checkout-7d9f8b6c4-x2k9p -n shop
   # Look for:
   #   Last State:  Terminated
   #     Reason:    OOMKilled | Error | Completed
   #     Exit Code: 137 | 1 | 0 ...

3. 3

   Pull the logs of the crashed instance

   The current instance is usually still starting and has nothing to say. The previous one has the stack trace.

   kubectl logs checkout-7d9f8b6c4-x2k9p -n shop -c checkout --previous

4. 4

   Read the events for this Pod

   Events tell you whether the kubelet killed the container (failed liveness probe) or the app exited on its own. Look for “Back-off restarting failed container” and probe failures.

   kubectl get events -n shop \
     --field-selector involvedObject.name=checkout-7d9f8b6c4-x2k9p \
     --sort-by=.lastTimestamp

5. 5

   Check what changed

   Most loops start right after a rollout. Compare the current revision with the last one that worked.

   kubectl rollout history deployment/checkout -n shop
   kubectl rollout history deployment/checkout -n shop --revision=12

6. 6

   Get a shell in a copy of the Pod

   You can’t exec into a container that dies in two seconds. Create a copy with the command swapped for a shell, then run the real entrypoint by hand and watch it fail.

   kubectl debug checkout-7d9f8b6c4-x2k9p -n shop -it \
     --copy-to=checkout-debug --container=checkout -- sh
   # Clean up afterwards
   kubectl delete pod checkout-debug -n shop

7. 7

   Fix the spec, roll it out, verify

   Change the thing the evidence points at (config, limit, probe, command), ship it through your normal pipeline, and confirm the restart count stops climbing.

   kubectl rollout status deployment/checkout -n shop --timeout=5m
   kubectl get pods -n shop -l app=checkout -w

A note on step 6. The kubectl debug reference documents both the --copy-to flag (“Create a copy of the target Pod with this name”) and ephemeral debug containers with --target. For a crash loop, the copy is usually the better move: an ephemeral container can only share processes with a target that’s actually running, and yours keeps dying. If the image has no shell, add --image=busybox and --share-processes to attach a toolbox container to the copy instead.

Hot take: most crash loops are solved by steps 2 and 3 alone. If you only remember one command from this post, make it kubectl logs --previous.

Decode the exit code: 137, 1, 127 and friends

The exit code in Last State is your fastest triage signal. Codes above 128 follow the usual Unix convention: 128 plus the signal number that killed the process.

Want the reason and code without scrolling through describe output? This prints the last termination for every container in the Pod:

kubectl get pod checkout-7d9f8b6c4-x2k9p -n shop \
  -o jsonpath='{range .status.containerStatuses[*]}{.name}{": "}{.lastState.terminated.reason}{" exit="}{.lastState.terminated.exitCode}{"\n"}{end}'

Exit code 1 with empty previous logs is its own vibe check: the app is probably failing before its logger initializes. Run the entrypoint by hand in a debug copy (step 6) and you’ll see the raw error on stderr. Searching thousands of log lines for the first real error is where asking your logs questions in plain English saves real time.

The 6 most common CrashLoopBackOff causes and their fixes

1. The app crashes on boot (exit code 1)

Bad environment variable, malformed config file, a schema migration that fails, a required flag that got renamed. The previous logs will say so, loudly. Fix the config, not the Pod. If it started right after a deploy, see the rollback section below before you start a heroic debugging session.

2. OOMKilled: exit code 137 and the memory limit

Reason OOMKilled means the container went over its memory limit and the kernel killed it. Check the limit against what the process actually needs at peak, including JVM heap plus off-heap, or your runtime’s equivalent:

resources:
  requests:
    memory: 512Mi
  limits:
    memory: 1Gi   # was 256Mi, below the app's startup peak

If usage climbs steadily until it hits any limit you set, you have a leak, and raising the limit only buys time. That’s a pattern you can spot hours before the kill: HealR’s Predictive Intelligence forecasting watches for memory leaks and resource exhaustion trends.

3. A liveness probe kills the app before it’s ready

The app is fine, it just takes 90 seconds to warm up, and the liveness probe gives it 30. The probes documentation is explicit: if a container fails its liveness probe more times than the configured tolerance, the kubelet restarts it, and liveness probes don’t wait for readiness. The tell is in the events: “Liveness probe failed” right before each restart, often with exit code 143 or 137. The fix is a startup probe, which holds off liveness and readiness checks until the app is up:

startupProbe:
  httpGet:
    path: /healthz
    port: 8080
  periodSeconds: 5
  failureThreshold: 30   # up to 150s to start
livenessProbe:
  httpGet:
    path: /healthz
    port: 8080
  periodSeconds: 10
  failureThreshold: 3

And keep the liveness endpoint about the process itself. A liveness check that pings your database turns one slow database into every Pod restarting at once. No cap, that’s how “self-healing” becomes a self-inflicted outage.

4. A dependency isn’t reachable at startup

The app tries to connect to Postgres, Redis or an upstream API on boot, fails, and exits. Logs show connection refused, DNS failures or timeouts. Check the dependency’s own health, its Service and endpoints, and any NetworkPolicy in the way. Long term, make the app retry with backoff instead of exiting on the first failed connection. Knowing what your service actually depends on is half this fight, which is exactly what TopologyGPT’s service dependency mapping is for.

5. Wrong command, args or architecture (exit 127, 126, 139)

A typo in command, an entrypoint that moved when the base image changed, a script missing its executable bit, or an image built for amd64 landing on an arm64 node. Run kubectl debug with --copy-to and try the command by hand; the error is usually instant.

6. The container exits 0 and gets restarted anyway

Reason Completed, exit code 0, still looping. Under restartPolicy: Always, “successfully finished” still means “restart it.” Either the process should stay in the foreground (a server that daemonized itself, a shell script that backgrounds the real process and exits), or the workload is really a batch task and belongs in a Job.

Same checklist applies to Init:CrashLoopBackOff: add -c with the init container’s name to your logs command.

Stop the bleeding: roll back first, debug second

If the loop started with a rollout and users are feeling it, your first job is mitigation, not root cause. A Deployment won’t undo itself, but one command will:

kubectl rollout undo deployment/checkout -n shop
kubectl rollout status deployment/checkout -n shop --timeout=5m

Two warnings. First, if you deploy through GitOps, a manual undo gets overwritten on the next sync, so revert the commit too. Second, deleting the Pod is not a fix: the controller recreates it from the same spec and it crashes the same way.

Before any change in production, know who’s downstream. A rollback that fixes one service can break the three that already adopted its new API. HealR’s blast radius analysis calculates the impact of a failure across your infrastructure, so “should we roll back?” becomes a data question.

Alert on CrashLoopBackOff without paging on every restart

A single restart is Kubernetes doing its job. A Pod stuck in the loop for 15 minutes is a human’s job. With kube-state-metrics running, page on the second and leave the first alone:

groups:
  - name: pod-crashloops
    rules:
      - alert: PodCrashLooping
        expr: |
          max by (namespace, pod, container) (
            kube_pod_container_status_waiting_reason{reason="CrashLoopBackOff"}
          ) == 1
        for: 15m
        labels:
          severity: page
        annotations:
          summary: "{{ $labels.namespace }}/{{ $labels.pod }} ({{ $labels.container }}) is crash looping"

      - alert: ContainerOOMKilledRepeatedly
        expr: |
          sum by (namespace, pod, container) (
            increase(kube_pod_container_status_restarts_total[1h])
          ) > 3
          and on (namespace, pod, container)
          kube_pod_container_status_last_terminated_reason{reason="OOMKilled"} == 1
        labels:
          severity: ticket

The second rule routes repeated OOM kills to a ticket rather than a page: it’s a sizing problem that rarely needs someone out of bed. For the bigger picture on cutting pager noise, read our guide on how to reduce alert fatigue for on-call teams.

Prefer a panel to a rule? Restart counts by namespace, top crash-looping containers and OOM kills over time make a solid “is anything looping?” dashboard, and you can describe it instead of hand-writing PromQL.

How AI agents speed up CrashLoopBackOff triage

Look at the seven steps again. Steps 1 to 5 are pure evidence gathering: the same commands, in the same order, on every crash loop, usually by someone who was asleep four minutes ago. That’s textbook toil, and it’s exactly the part software should do.

HealR’s Agentic War Room for Kubernetes incident investigation runs that loop with specialized agents: a Metrics Agent and a Logs Agent gather the signals, a correlation step lines them up on one timeline, an RCA step ranks root-cause hypotheses with confidence scores, and a recommendations step proposes remediation with a risk assessment. Every conclusion comes with the evidence behind it, so you check the homework instead of doing it.

What it shouldn’t do is quietly bump memory limits in prod at 3 AM. HealR goes live with human-in-the-loop guardrails by default, and autonomy widens only as trust builds. We covered where that line sits in self-healing Kubernetes with AI agents.

FAQ: fixing CrashLoopBackOff