Debug a Failing Pod: describe, logs, exec, and port-forward
A repeatable checklist for finding out why a Pod is Pending, crashing, or not answering.
Sooner or later a Pod won’t start, keeps restarting, or runs but doesn’t respond. The fix is almost always visible from a handful of kubectl commands, used in the right order. This guide gives you that order: read the status, ask for events, read the logs, get inside the container, and then trace the network path.
Read the status first
Start with the status column:
kubectl get pods -o wide
The most common states, and where to look first:
- Pending. The Pod isn’t assigned to a node. Usually there isn’t enough CPU or memory, a quota is exhausted, or a volume can’t bind.
describewill say which. - ContainerCreating. The node is preparing the container. If it stays this way, check for a missing volume or a slow image pull.
- ErrImagePull / ImagePullBackOff. The image can’t be pulled, most often because of a typo in the name or tag, or missing credentials for a private registry.
- CrashLoopBackOff. The container starts, exits, and is restarted repeatedly with growing delays. The application is failing, so read its logs.
- CreateContainerConfigError. The Pod references a ConfigMap, Secret, or key that doesn’t exist.
- Running but
READY 0/1. The container is up but failing its readiness probe. - OOMKilled (shown in
describe). The container exceeded its memory limit.
Ask Kubernetes why: describe and events
kubectl describe pod POD_NAME
Scroll to the Events section at the bottom. It reads like a timeline: Scheduled, Pulling, Failed, BackOff. Messages such as 0/3 nodes are available: insufficient cpu or Failed to pull image "ngnix:1.25" usually name the problem outright. Higher up, look at each container’s State and Last State for exit codes and reasons. For a namespace-wide view of what has been happening:
kubectl get events --sort-by=.metadata.creationTimestamp
Read the container’s output
kubectl logs POD_NAME
kubectl logs POD_NAME -c CONTAINER_NAME
kubectl logs POD_NAME --all-containers
kubectl logs POD_NAME --previous
kubectl logs POD_NAME -f --since=10m --tail=100
Use -c when a Pod has several containers, -f to stream, --since and --tail to limit the output, and --previous to read the logs of the last crashed instance, which is what you need for a crash loop. To try it, create a Deployment whose container fails on purpose:
kubectl create deployment crashy --image=busybox:1.36 -- \
sh -c 'echo starting; echo "fatal: DB_URL is not set" >&2; exit 1'
kubectl get pods -l app=crashy
kubectl logs deployment/crashy --previous
The status becomes CrashLoopBackOff, and the log shows why. Clean up with kubectl delete deployment crashy.
Keep in mind that logs belong to the Pod. When it is deleted or replaced, its logs are cleaned up with it, so for history you need a log collector such as Fluentd or Fluent Bit shipping logs somewhere durable.
Get inside: exec, port-forward, and cp
kubectl exec -it POD_NAME -- sh
kubectl exec POD_NAME -- env
kubectl port-forward pod/POD_NAME 8080:80
curl http://localhost:8080
kubectl cp POD_NAME:/path/to/file ./file
exec runs a command in the container, port-forward connects a local port to a port in the Pod (it works for svc/NAME too), and cp copies files in or out. Many production images are minimal and have no shell or curl. In that case kubectl debug -it POD_NAME --image=busybox:1.36 --target=CONTAINER_NAME attaches a temporary debugging container that shares the Pod’s namespaces.
Anything you change inside a running container, like editing a config file, is lost as soon as the Pod is replaced. Use exec to investigate, but make the fix in the image, the manifest, or a ConfigMap.
Work backwards through the request path
If the Pod looks healthy but the application isn’t reachable, trace the path from the inside out:
- Is the Pod running and
READY? Fix the Pod first. - Does the Service selector match the Pod’s labels?
kubectl get pods --show-labelsandkubectl describe service NAMEshould agree. - Does the Service have endpoints?
kubectl get endpointslices -l kubernetes.io/service-name=NAMEshould list Pod IPs. An empty list means a selector mismatch or no ready Pods. - Do the ports match? The Service’s
targetPortmust be the port the container actually listens on. - Does it work from another Pod?
kubectl run tmp --rm -it --image=busybox:1.36 --restart=Never -- wget -qO- http://NAME. This tests DNS and the Service, ruling out anything outside the cluster. - Could a NetworkPolicy be blocking it? List them with
kubectl get networkpolicy. - Finally, check the Ingress or external load balancer in front of it.
Each step narrows the problem to one layer, which is faster than guessing.
Popular Guides
-
What Kubernetes Is and the Problems It Solves Why containers alone stop scaling, and how Kubernetes turns a group of machines into one platform. -
How a Kubernetes Cluster Works: Control Plane and Nodes The components that make a cluster tick, and what happens between kubectl apply and a running container. -
Deployments: Scaling, Self-Healing, and Rolling Updates Run replicas of a Pod, update them without downtime, and roll back when a release goes wrong. -
Services: Stable Networking for Ephemeral Pods Give Pods a stable address, expose them inside and outside the cluster, and restrict who can talk to whom.