Skip to content
Creed Wiki

CrashLoopBackOff and pod states

In one line

Section titled “In one line”

CrashLoopBackOff means a container started, exited, and the kubelet is now waiting before it starts it again. The wait doubles from 10 seconds up to 5 minutes. It is a symptom, not a cause: the cause is in the exit code, the previous logs and the events.

Why it exists

Section titled “Why it exists”

A container that dies must come back, or a single crash would take a service down for good. That is the job of restartPolicy. But a container that dies at once, every time, would be restarted hundreds of times a minute. Each restart costs an image check, a sandbox setup, a log file and API writes, so one broken pod could load the node and the API server.

The kubelet therefore restarts with an exponential back-off: quick at first, in case the failure was a one-off, then slower. The name describes the state exactly: the container is in a crash loop, and the kubelet is in the back-off part of it.

Two things confuse people:

  • CrashLoopBackOff is not a pod phase. The phases are Pending, Running, Succeeded, Failed and Unknown. A crash-looping pod is in phase Running. What kubectl get pods prints in the STATUS column is a display string built from the container state (Waiting, with reason CrashLoopBackOff).
  • The container is not running while you look at it. That is why kubectl logs often shows nothing useful and kubectl exec fails. You need the logs of the previous instance.

How it works

Section titled “How it works”

The restart loop

Section titled “The restart loop”
  1. The kubelet starts the container.
  2. The process exits. Any exit counts when restartPolicy is Always, including exit code 0.
  3. The first restart is immediate.
  4. On repeated crashes the kubelet waits before each restart: 10s, 20s, 40s, 80s, 160s, then 300s. The delay doubles each time and is capped at 300 seconds (5 minutes). While it waits, the container state is Waiting with reason CrashLoopBackOff.
  5. Once a container has run for 10 minutes without a problem, the kubelet resets the back-off timer for that container. The next crash is treated as the first one.

The restart happens in place: same pod, same node, same IP, same volumes. RESTARTS in kubectl get pods counts it. A crash loop never moves the pod to another node.

THE LOOP kubelet starts container process exits Waiting: CrashLoopBackOff wait the back-off delay, then start again DELAY BEFORE EACH RESTART (DEFAULT) cap: 300s (5 min) 10s 20s 40s 80s 160s 300s 300s delay 1 delay 2 delay 3 delay 4 delay 5 delay 6 delay 7+ The timer resets once the container has run 10 minutes without a problem.
After five doublings every further restart waits the full 5 minutes, so after you fix a missing dependency the pod can sit for up to 5 minutes before it tries again. Deleting the pod skips the wait.

Tuning the back-off (version-specific)

Section titled “Tuning the back-off (version-specific)”
Feature gate Stage in Kubernetes 1.37 What it changes
none (default) stable behaviour start at 10s, double, cap at 300s, reset after 10 minutes of clean running
KubeletCrashLoopBackOffMax alpha in 1.32, beta and on by default since 1.35 per node cap: set crashLoopBackOff.maxContainerRestartPeriod in the kubelet configuration, between "1s" and "300s". Delays still start at 10s. If the cap is below 10s, the initial delay becomes the cap
ReduceDefaultCrashLoopBackOffDecay alpha since 1.33, off by default start at 1s instead of 10s, cap at 60s instead of 300s, for the whole node 
# kubelet configuration: back-off still starts at 10s, but never exceeds 100s
kind: KubeletConfiguration
crashLoopBackOff:
  maxContainerRestartPeriod: "100s"

Image pulls have their own back-off with the same compiled-in cap of 300 seconds. That one shows as ImagePullBackOff.

restartPolicy

Section titled “restartPolicy”

spec.restartPolicy is set per pod and applies to its app containers.

Value Restarts when Used by Can it crash loop
Always (default) the container exits for any reason, exit code 0 included Deployment, StatefulSet, DaemonSet (they accept only Always) yes
OnFailure the container exits with a non-zero code Job, CronJob yes, until the Job’s backoffLimit fails the Job
Never never Job, one-off pods no. The pod goes to phase Failed and shows Error

Details that change the picture:

  • Exit 0 under Always still loops. A container whose process finishes its work and exits cleanly shows Completed, then CrashLoopBackOff. The usual cause is a one-shot command in a Deployment, or a process that forks into the background and lets PID 1 exit.
  • Init containers run one by one before the app containers. If one fails, the kubelet retries it under the pod’s restartPolicy (with Always it behaves as OnFailure). The status reads Init:Error, then Init:CrashLoopBackOff. The app containers never start.
  • Sidecar containers are init containers with their own restartPolicy: Always (stable since 1.33). They start before the app containers, keep running, and restart on their own regardless of the pod-level policy.
  • Per-container restart rules (ContainerRestartRules, beta and on by default since 1.35): a container can set its own restartPolicy and restartPolicyRules that match on exit codes. The back-off is the same.
  • A crash loop is not a reschedule. Only deleting the pod (or the node failing) makes a controller create a new pod, possibly on another node.

Probes

Section titled “Probes”

Probes are the second way a container gets restarted. The kubelet runs them; they do not depend on the process exiting.

Startup probe Liveness probe Readiness probe
Question has the application finished starting is the application still working can it take traffic now
On failure the kubelet kills the container; restartPolicy decides what happens next the kubelet kills the container; restartPolicy decides what happens next the pod is removed from Service endpoints. No restart
Runs from container start until it first succeeds. Liveness and readiness do not run until then for the whole life of the container, after the startup probe has passed for the whole life of the container, after the startup probe has passed
Shows as RESTARTS grows, CrashLoopBackOff, event Startup probe failed RESTARTS grows, CrashLoopBackOff, event Liveness probe failed then Killing READY 0/1, pod stays Running, event Readiness probe failed
Use it for slow starters: give them failureThreshold * periodSeconds to come up deadlocks that only a restart fixes warm-up, temporary overload, a dependency that is down
Classic mistake missing, so a tight liveness probe kills a slow start forever checking a database or another service, so one outage restarts every replica missing, so traffic arrives before the application listens

Defaults for every probe: initialDelaySeconds 0, periodSeconds 10, timeoutSeconds 1, failureThreshold 3, successThreshold 1 (and it must be 1 for liveness and startup). So a default liveness probe kills a container after about 30 seconds of failed checks, and a probe handler that takes longer than 1 second counts as a failure.

Probe mechanisms: httpGet (any status from 200 to 399 passes), tcpSocket, grpc, exec (exit 0 passes; each run forks a process in the container).

Triage path

Section titled “Triage path”

Work in this order. Each step answers one question.

  1. Which container, how often, since when.

    Terminal window
    kubectl get pod web-7d4b9c6f5-x2kqp -o wide
    NAME                  READY   STATUS             RESTARTS        AGE   NODE
    web-7d4b9c6f5-x2kqp   0/1     CrashLoopBackOff   6 (2m10s ago)   9m    ip-10-0-3-17

  2. Why did the last instance end. Look at Last State, Reason, Exit Code and the events at the bottom.

    Terminal window
    kubectl describe pod web-7d4b9c6f5-x2kqp
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       OOMKilled
      Exit Code:    137
      Started:      Thu, 12 Nov 2026 10:01:07 +0000
      Finished:     Thu, 12 Nov 2026 10:01:09 +0000
    Restart Count:  6
    Events:
      Warning  BackOff  2m (x25 over 8m)  kubelet  Back-off restarting failed container web in pod web-7d4b9c6f5-x2kqp

  3. What did it say before it died. The current instance has not started yet, so ask for the previous one.

    Terminal window
    kubectl logs web-7d4b9c6f5-x2kqp -c web --previous
    kubectl logs web-7d4b9c6f5-x2kqp -c init-db          # an init container by name

  4. Events for the pod, oldest first, when describe has already aged them out of view.

    Terminal window
    kubectl get events --field-selector involvedObject.name=web-7d4b9c6f5-x2kqp --sort-by=.lastTimestamp

  5. Read the exit code. See the table below.

  6. If logs are empty, the process died before it could write, or the container never ran: wrong command, wrong architecture, missing file, permissions. Go to “Keeping a crashing container alive”.

Exit codes. A code above 128 means the process was killed by a signal: the code is 128 plus the signal number.

Exit code Meaning Usual cause Next check
0 clean exit one-shot command under restartPolicy: Always, process daemonised itself the command and args; should this be a Job
1 application error unhandled exception, bad configuration, dependency refused the connection at start logs --previous
2 misuse of a shell builtin, or the application’s own choice bad flags, shell script error logs --previous, the entrypoint script
126 command found but not executable missing execute bit, wrong interpreter, read-only mount ls -l on the binary in a debug copy
127 command not found typo in command, binary not in the image, sh missing in a distroless image the image contents, command against ENTRYPOINT
137 SIGKILL (128 + 9) OOMKilled, or the kubelet killed it after a failed liveness or startup probe and the grace period ran out Reason in describe; memory limit against real usage
139 SIGSEGV (128 + 11) segmentation fault: native code bug, wrong libc (glibc binary on musl), bad binary for the CPU logs, the image’s base and architecture
143 SIGTERM (128 + 15) the kubelet asked it to stop: probe failure, pod deletion, node drain. The process exited on the signal events for Killing and Unhealthy

Causes that produce a crash loop, and how each one looks:

Cause How it shows Check
Application bug or bad config value exit 1, stack trace in previous logs kubectl logs --previous
Out of memory Reason: OOMKilled, exit 137, often seconds after start kubectl describe pod, kubectl top pod, resources.limits.memory
Liveness or startup probe misconfigured events Liveness probe failed and Killing, exit 137 or 143, logs look healthy probe path, port, timeoutSeconds, add a startup probe
ConfigMap or Secret key missing (as env) not a crash loop: CreateContainerConfigError, the container never starts kubectl describe pod, events name the missing object
ConfigMap or Secret present but wrong content exit 1 with a parse or auth error kubectl get configmap app -o yaml, compare with what the application expects
Bad command or args exit 127 or 126, or StartError or RunContainerError with executable file not found in $PATH kubectl get pod -o jsonpath='{.spec.containers[0].command}'
Image built for another CPU architecture exec format error in logs or in the container status message kubectl get node -o wide for the node’s architecture, docker buildx imagetools inspect IMAGE
Permissions and securityContext permission denied, read-only file system, operation not permitted, bind to a port below 1024 as non-root runAsUser, runAsNonRoot, fsGroup, readOnlyRootFilesystem, dropped capabilities
Dependency not ready exit 1 with connection refused or DNS failure, heals by itself later the dependency, DNS from a debug container; add retries in the application rather than ordering pods
Init container failing Init:CrashLoopBackOff, app containers PodInitializing kubectl logs POD -c INIT_NAME --previous
Volume content missing or wrong owner exit 1 or 126 right after start kubectl debug with the volume, fsGroup

Keeping a crashing container alive

Section titled “Keeping a crashing container alive”

kubectl exec needs a running container, and a crash-looping one is not. Four ways in, from least to most invasive.

  1. Ephemeral container in the same pod (stable since Kubernetes 1.25). It adds a container with your tools to the live pod. --target joins the process namespace of the named container, which works only while that container is running, and the container runtime must support it. Ephemeral containers have no resource requests, no ports, no probes, are never restarted, and cannot be removed again.

    Terminal window
    kubectl debug -it web-7d4b9c6f5-x2kqp --image=busybox:1.36 --target=web --profile=general

  2. Copy of the pod with the command replaced. This is the one for a container that dies at startup. The copy keeps the image, environment and volumes, but runs a shell instead of the entrypoint, so nothing crashes.

    Terminal window
    kubectl debug web-7d4b9c6f5-x2kqp -it --copy-to=web-debug --container=web -- sh
    # inside: run the real entrypoint by hand and watch it fail
    kubectl delete pod web-debug

  3. Copy with another image, for images without a shell (distroless, scratch). --share-processes lets the debug container see the application’s processes and reach its filesystem through /proc/PID/root.

    Terminal window
    kubectl debug web-7d4b9c6f5-x2kqp -it --copy-to=web-debug --image=ubuntu --share-processes
    kubectl debug web-7d4b9c6f5-x2kqp --copy-to=web-debug --set-image=*=ubuntu

  4. Change the workload itself: set command: ["sleep", "infinity"] on the Deployment, and remove the liveness and startup probes so the kubelet does not kill the sleeping container. Use it on a non-production copy. With a GitOps controller the change will be reverted, see Admission control and GitOps drift.

kubectl debug profiles set the security context of the debug container: legacy (the default when --profile is not given, planned for deprecation), general, baseline, restricted, netadmin, sysadmin. Use --profile=restricted in namespaces that enforce the restricted Pod Security Standard, and --profile=sysadmin when you need root and all capabilities.

To look at the node instead of the pod:

Terminal window
kubectl debug node/ip-10-0-3-17 -it --image=ubuntu --profile=sysadmin
# the node's root filesystem is mounted at /host

Compare

Section titled “Compare”

Related pod states. STATUS is what kubectl get pods prints.

STATUS What it means Did the container run Usual cause First command
CrashLoopBackOff the container exited and the kubelet is waiting before the next restart yes, and it exited application error, OOM, probe kill, bad command kubectl logs POD --previous
Error the container exited non-zero. Under Always or OnFailure it turns into CrashLoopBackOff; under Never it stays yes same as above kubectl logs POD
Completed the container exited with code 0 yes normal for Jobs; under Always it is a one-shot command in the wrong kind of workload kubectl get pod POD -o jsonpath='{.spec.containers[*].command}'
OOMKilled the kernel killed the container because it passed its memory limit (exit 137) yes limit too low, leak, JVM or runtime not aware of the cgroup limit kubectl describe pod POD, kubectl top pod POD --containers
ErrImagePull one image pull attempt failed no wrong name or tag, no imagePullSecrets, registry unreachable, rate limit kubectl describe pod POD (events carry the registry error)
ImagePullBackOff pulls keep failing and the kubelet is waiting before the next try (cap 300s) no same as ErrImagePull kubectl describe pod POD
Pending the pod is accepted but not running yet: not scheduled, or still pulling and creating no no node fits (CPU, memory, taints, affinity), unbound PersistentVolumeClaim kubectl describe pod POD (event FailedScheduling)
ContainerCreating scheduled, the kubelet is setting up sandbox, network and volumes no volume mount waiting (missing Secret or ConfigMap volume, CSI attach), CNI failure kubectl describe pod POD (events FailedMount, FailedCreatePodSandBox)
CreateContainerConfigError the kubelet cannot build the container’s configuration no env valueFrom points at a ConfigMap, Secret or key that does not exist kubectl describe pod POD
Init:CrashLoopBackOff an init container keeps failing; app containers have not started only the init container migration fails, dependency check never passes kubectl logs POD -c INIT_NAME --previous
Init:N/M N of M init containers have finished; the next one is running init containers only slow or blocked init step kubectl logs POD -c INIT_NAME
Evicted the kubelet ended the pod to protect the node. Phase Failed, the pod object stays behind yes, until evicted node pressure: memory, disk or ephemeral storage, PIDs kubectl describe pod POD (message names the resource), kubectl describe node NODE
Terminating (stuck) the pod has a deletionTimestamp but is still there usually still running a finalizer that nobody clears, a node that is NotReady, a process that ignores SIGTERM with a long grace period kubectl get pod POD -o jsonpath='{.metadata.finalizers}'
Unknown the control plane cannot reach the kubelet on the node unknown node down or partitioned kubectl get node, kubectl describe node NODE

Who restarts what.

Event Who acts Result
Container exits kubelet, per restartPolicy same pod, same node, RESTARTS + 1, back-off
Liveness or startup probe fails kubelet container killed, then as above
Readiness probe fails kubelet reports, endpoint controllers act pod leaves the Service endpoints, no restart
Pod deleted or evicted the owning controller (ReplicaSet, StatefulSet, Job) a new pod, new name (except StatefulSet), maybe another node, no back-off carried over
Node lost node lifecycle controller, then the owning controller pods evicted after the toleration time (default 300s for not-ready and unreachable), new pods elsewhere

What breaks

Section titled “What breaks”
Symptom Cause Check
CrashLoopBackOff, logs empty, logs --previous empty the process never wrote anything: wrong binary, exec format error, missing library, or logs go to a file kubectl describe pod for the container status message; a debug copy with -- sh
CrashLoopBackOff with exit 0 the main process ends or backgrounds itself command and args; run the process in the foreground
Restarts every 30 to 40 seconds, logs look normal liveness probe fails: wrong port or path, 1s timeout too short, application still starting events Unhealthy; add a startup probe, raise timeoutSeconds
All replicas restart together during a dependency outage liveness probe checks the dependency move that check to the readiness probe
OOMKilled within seconds of start limit below the runtime’s baseline, or heap sized from node memory kubectl describe pod; the runtime’s heap flags against limits.memory
Works on one node pool, crash loops on another CPU architecture or kernel feature differs kubectl get nodes -L kubernetes.io/arch
Fix deployed, pod still waits the back-off timer is at up to 300s kubectl delete pod POD so the controller creates a fresh one
kubectl exec says the container is not running it is between restarts kubectl debug with --copy-to
Init:CrashLoopBackOff after a deploy init step (migration, wait-for) fails kubectl logs POD -c INIT_NAME --previous
Pod Terminating for hours finalizer, or the node is gone finalizers on the pod; node status. Force removal only when you know the container is not running: kubectl delete pod POD --grace-period=0 --force
Pod Running, READY 0/1, no restarts readiness probe fails; not a crash events Readiness probe failed, the probe target
On AWS the application starts, then dies with access denied the pod has no usable AWS identity EKS Pod Identity and IRSA

Useful one-liners:

Terminal window
# reason, exit code and times of the last termination, per container
kubectl get pod POD -o jsonpath='{range .status.containerStatuses[*]}{.name}{"\t"}{.lastState.terminated.reason}{"\t"}{.lastState.terminated.exitCode}{"\t"}{.lastState.terminated.finishedAt}{"\n"}{end}'


# every pod that is not healthy, across namespaces
kubectl get pods -A --field-selector=status.phase!=Running,status.phase!=Succeeded
kubectl get pods -A | grep -Ev 'Running|Completed'


# restart counts, highest last
kubectl get pods -A --sort-by='.status.containerStatuses[0].restartCount'


# what the container actually runs
kubectl get pod POD -o jsonpath='{.spec.containers[0].image}{"\n"}{.spec.containers[0].command}{"\n"}{.spec.containers[0].args}{"\n"}'


# watch the loop live
kubectl get pod POD -w

Keywords

Section titled “Keywords”

CrashLoopBackOff, back-off, exponential back-off, restartPolicy, Always, OnFailure, Never, pod phase, container state, Waiting, Terminated, RESTARTS, kubectl logs –previous, kubectl describe, events, exit code, 137, 139, 143, 126, 127, SIGKILL, SIGTERM, SIGSEGV, OOMKilled, exec format error, liveness probe, readiness probe, startup probe, failureThreshold, periodSeconds, timeoutSeconds, init container, Init:CrashLoopBackOff, sidecar container, ImagePullBackOff, ErrImagePull, CreateContainerConfigError, ContainerCreating, Pending, FailedScheduling, Evicted, node pressure, Terminating, finalizer, kubectl debug, ephemeral container, –copy-to, –share-processes, –target, debug profile, KubeletCrashLoopBackOffMax, maxContainerRestartPeriod, ReduceDefaultCrashLoopBackOffDecay, ContainerRestartRules.

Sources

Section titled “Sources”