Kubernetes Troubleshooting Guide: Diagnose Pods, Services, Ingress, and Cluster Failures

Overview

Kubernetes failures are often symptoms of a problem in a different layer. A pod may be restarting because of an application error, a failed probe, insufficient memory, a missing secret, or a dependency that the pod cannot reach. A service may have no usable endpoints even though its selector appears correct. An ingress may be healthy inside the cluster while external requests fail at the controller, load balancer, DNS, or TLS layer.

The fastest approach is to move from broad observations to narrow evidence. Start with the workload and namespace, inspect recent events, check the pod status and container logs, then follow the request path through services, endpoints, ingress, and external infrastructure. Avoid changing several objects at once: a clean diagnosis is easier when each test has a clear purpose.

Set a namespace explicitly in commands when possible. This reduces the risk of inspecting one environment while assuming you are looking at another.

kubectl config current-context
kubectl get namespaces
kubectl get pods -n <namespace>
kubectl get events -n <namespace> --sort-by=.lastTimestamp

For broader platform work, keep this guide alongside your normal Kubernetes tools, observability dashboards, deployment records, and runbooks. The goal is not to memorize every kubectl command; it is to build a reliable path from symptom to evidence to corrective action.

What to track

1. Pod state and restart patterns

Begin with the workload summary and then inspect the pod that shows an unusual state.

kubectl get pods -n <namespace> -o wide
kubectl describe pod <pod-name> -n <namespace>
kubectl logs <pod-name> -n <namespace> --all-containers
kubectl logs <pod-name> -n <namespace> --previous

Track whether the pod is Pending, repeatedly restarting, failing readiness, or being terminated soon after startup. In describe output, pay attention to the events section, container state, exit reason, image details, mounted volumes, environment references, and probe configuration.

Common interpretations include:

• Pending: the scheduler may be unable to find a suitable node, a required volume may not be available, or a scheduling constraint may be too restrictive.
• CrashLoopBackOff: the container starts and exits repeatedly. Inspect current and previous logs, exit codes, startup configuration, and dependency connectivity.
• ImagePullBackOff or ErrImagePull: check the image name and tag, registry access, image pull credentials, and whether the node can reach the registry.
• NotReady: determine whether the readiness probe is failing, the application is still starting, or a required dependency is unavailable.
• OOMKilled: compare the container's memory behavior with its configured limit and investigate leaks, workload changes, or unexpectedly large requests.

2. Events, probes, and recent changes

Events provide time-ordered clues, but they are not a complete incident history. Capture them while the problem is active and correlate them with deployment, configuration, and infrastructure changes.

kubectl get events -A --sort-by=.lastTimestamp
kubectl describe deployment <deployment-name> -n <namespace>
kubectl rollout history deployment/<deployment-name> -n <namespace>
kubectl rollout status deployment/<deployment-name> -n <namespace>

For probes, check the path, port, scheme, initial delay, timeout, and failure threshold. A liveness probe should not be used as a substitute for dependency-aware readiness. If an application needs time to initialize, review whether a startup probe or a more conservative initial configuration is appropriate. Change probe settings only after confirming that the probe reflects the intended health signal.

3. Services, selectors, and networking

When a service does not route traffic, verify the complete chain: service selector, pod labels, endpoints, target port, and application listener.

kubectl get svc <service-name> -n <namespace> -o yaml
kubectl get endpoints <service-name> -n <namespace>
kubectl get endpointslices -n <namespace> -l kubernetes.io/service-name=<service-name>
kubectl get pods -n <namespace> --show-labels

No endpoints usually means that the selector does not match ready pods, the pods are not ready, or the service is pointing at the wrong labels. If endpoints exist, test connectivity from a suitable pod or diagnostic container and compare the service port with the container port. Also check network policies, DNS resolution, and any service mesh configuration before concluding that the application itself is unavailable.

4. Ingress, TLS, and external access

For ingress failures, separate internal routing from external exposure. Inspect the ingress object, controller logs, service, backend endpoints, and TLS secret.

kubectl get ingress -n <namespace>
kubectl describe ingress <ingress-name> -n <namespace>
kubectl get svc -n <controller-namespace>
kubectl logs -n <controller-namespace> deploy/<controller-deployment>

Check the requested host and path, ingress class, backend service port, certificate secret, controller address, and DNS record. A default backend response, a timeout, a certificate warning, and a connection refusal point to different layers. The relevant controller-specific syntax varies, so confirm the installed controller's conventions rather than copying configuration from an unrelated cluster.

5. Resources, storage, and cluster health

Resource pressure can look like an application failure. Review requests and limits, node conditions, scheduling decisions, and recent evictions.

kubectl top pods -A
kubectl top nodes
kubectl get nodes
kubectl describe node <node-name>
kubectl get pvc -A

For storage issues, inspect the persistent volume claim, storage class, binding state, mount events, and access mode. For node problems, look for pressure conditions, taints, capacity changes, and workloads concentrated on a small number of nodes. Treat destructive actions such as deleting a pod, forcing a rollout, or removing a finalizer as deliberate operations with an understood recovery path.

Cadence and checkpoints

Troubleshooting is more effective when recurring checks are scheduled instead of performed only during an incident. Use three practical cadences.

During every incident

• Record the cluster, context, namespace, affected workload, first observed time, and user-visible symptom.
• Capture pod descriptions, recent events, current and previous logs, rollout state, and relevant service or ingress details.
• Note what changed immediately before the failure: deployment, image, configuration, secret, node, DNS, certificate, or policy.
• Separate mitigation from root-cause work. Restoring service does not remove the need to preserve evidence.

Monthly operational review

Review restart counts, failed rollouts, pending pods, probe failures, evictions, persistent volume alerts, ingress errors, and recurring event messages. Look for the same workload or node appearing repeatedly. Compare these observations with deployment history and incident records. A small table with date, symptom, affected object, suspected layer, action, and outcome is enough to reveal patterns.

Quarterly platform review

Review cluster version support assumptions, add-on and controller compatibility, resource requests and limits, network policies, storage classes, access permissions, image provenance, and backup or restore procedures. Validate that runbooks still match the cluster's actual installation. Review access with the same care given to operational reliability; the Kubernetes RBAC best practices guide provides a useful companion for role and service-account reviews.

Use deployment and infrastructure records when investigating drift. The infrastructure drift detection guide can complement a Kubernetes review by helping distinguish declared configuration from changes made directly in a live environment.

How to interpret changes

Interpret trends by connecting symptoms to layers rather than counting alerts in isolation. A rise in restarts after a new image suggests an application or packaging path; a rise in pending pods after a capacity change suggests scheduling or resource constraints; a rise in ingress errors with healthy backend endpoints shifts attention toward the controller, load balancer, DNS, or TLS configuration.

Use a simple hypothesis loop:

1. State the symptom precisely. For example, “requests to one host return a timeout” is more useful than “ingress is broken.”
2. Choose the narrowest confirming command. Inspect the relevant object and its immediate dependency before changing configuration.
3. Compare healthy and unhealthy examples. Compare pods, nodes, namespaces, routes, or releases to find the meaningful difference.
4. Apply one controlled change. Record the change and observe whether the expected signal improves.
5. Document the durable fix. Update manifests, policies, alerts, or runbooks so the same diagnosis is easier next time.

Be cautious with noisy evidence. A warning event may be harmless in context, while a single failed readiness probe may explain an outage. Check timestamps, affected objects, and user impact. Avoid treating a successful command as proof that the entire request path is healthy; a pod can be running while its service, ingress, dependency, or external identity configuration remains broken.

When to revisit

Return to this guide whenever a recurring failure changes shape, a new controller or storage integration is introduced, or a deployment pattern is redesigned. Revisit it after incidents that expose an undocumented dependency, unclear ownership, or missing telemetry. It should also be part of monthly operational reviews and quarterly platform maintenance.

Make the next review actionable:

• Choose one recurring symptom, such as restarts, pending pods, probe failures, or ingress errors.
• Identify the first evidence you want responders to collect and turn it into a short runbook.
• Define the owner, escalation path, and safe mitigation for that symptom.
• Check that logs, events, metrics, and deployment history cover the time window needed for diagnosis.
• Test the runbook in a non-production environment or during a controlled maintenance window.

For related operational work, connect Kubernetes checks with CI/CD pipeline best practices, image hygiene from the Docker image optimization checklist, and appropriate security gates from the DevSecOps checklist. A troubleshooting guide is most valuable when its findings feed back into safer builds, clearer deployments, and more observable cloud-native infrastructure.