# OpenShift & Kubernetes — Production Troubleshooting Cheatsheet

By Komlan Florient DOGBE — Senior Cloud & SRE Engineer (RHCA, RHCOA, CKA, CKAD).
Commands and reflexes used day-to-day to diagnose a production cluster.

## 1. First look at the cluster

```bash
# Overall node status
kubectl get nodes -o wide
oc get nodes -o wide                      # OpenShift equivalent

# Resource pressure per node
kubectl top nodes
kubectl describe node <node> | grep -A5 "Allocated resources"

# Recent events (often the first thing to check)
kubectl get events -A --sort-by='.lastTimestamp' | tail -30
```

## 2. Pods that won't start

```bash
kubectl get pods -n <namespace> -o wide
kubectl describe pod <pod> -n <namespace>       # Events at the bottom of the output
kubectl logs <pod> -n <namespace> --previous     # logs from the previous container (crashloop)
kubectl logs <pod> -n <namespace> -c <container> # multi-container pods
```

Common causes to check in order:
1. `ImagePullBackOff` → missing/expired registry secret (`kubectl get secret -n <ns>`), or a tag that doesn't exist.
2. `CrashLoopBackOff` → `kubectl logs --previous`, usually a config/missing env var issue.
3. `Pending` → insufficient resources (`kubectl describe pod` → Events section → `FailedScheduling`), an unbound `PersistentVolumeClaim`, or mismatched `nodeSelector`/`taint`.
4. `OOMKilled` → `kubectl describe pod` → `Last State: Terminated, Reason: OOMKilled` → revisit `resources.limits.memory`.

## 3. Networking

```bash
# A pod can't reach a service
kubectl exec -it <pod> -n <namespace> -- sh
# then, inside the pod:
nslookup <service>.<namespace>.svc.cluster.local
curl -v http://<service>.<namespace>:<port>/health

# Check the actual endpoints behind a Service
kubectl get endpoints <service> -n <namespace>

# NetworkPolicy in place on the namespace
kubectl get networkpolicy -n <namespace> -o yaml
```

If `endpoints` is empty: the Service's `selector` doesn't match any pod — check the labels.

## 4. OpenShift specifics

```bash
# Routes
oc get routes -n <namespace>
oc describe route <route> -n <namespace>

# Build & deployment
oc get builds -n <namespace>
oc logs -f bc/<buildconfig> -n <namespace>
oc rollout status dc/<deploymentconfig> -n <namespace>

# Security / SCC (Security Context Constraints)
oc get scc
oc describe scc restricted
oc adm policy who-can use scc anyuid
```

## 5. Ansible Automation Platform — diagnosing a failed job

```bash
# From the controller
awx-cli job list --status failed
awx-cli job stdout <job_id>

# Execution Environment: reproduce locally
podman run --rm -it <ee-image> ansible --version
ansible-playbook -i inventory playbook.yml -vvv   # max verbosity
```

## 6. GitOps (ArgoCD)

```bash
argocd app get <app-name>
argocd app diff <app-name>          # what diverges between Git and the cluster
argocd app sync <app-name> --dry-run
argocd app logs <app-name> --follow
```

Persistent `OutOfSync` despite a sync → often a third-party controller (HPA, admission webhook) modifying the resource afterward; consider `ignoreDifferences` in the `Application`.

## 7. Etcd / API server (cluster-admin)

```bash
# Etcd health (on a master, or via oc debug node)
oc get etcd -o yaml
oc adm top images

# Certificates close to expiry
oc get co | grep -v "True.*False.*False"   # ClusterOperators reporting an anomaly
```

## 8. Checklist before a production intervention

- [ ] `kubectl config current-context` — confirm you're on the right cluster.
- [ ] Backup/dry-run before any destructive `apply` (`--dry-run=client -o yaml`).
- [ ] `PodDisruptionBudget` in place if you're about to `drain`/do a rolling update.
- [ ] Check `replicas` and `maxUnavailable` before a rollout.
- [ ] Communicate the maintenance window if there's a contractual SLA.

---

*This resource is part of [Komlan Florient DOGBE](https://www.florientdogbe.me/)'s portfolio. Find more lessons learned in the [blog](../blog/index.html).*