Why Your Pod Is Stuck in Pending, Explained With a Restaurant Host Stand

You're the host at a busy restaurant on a Friday night. A party of four walks in. Before you seat them, you run a quick checklist in your head: which tables are big enough, which ones are reserved, which section they asked for, and who they shouldn't be seated next to. If no table passes, they go on the waiting list. That is the Kubernetes scheduler. Every pod is a party, every node is a table, and Pending is the waiting list. Once you see it this way, the event at the bottom of kubectl describe pod reads like a host explaining why there's no table for you. (I used AI tools to help research and draft this post, and checked the Kubernetes behavior against the official docs.) The party size is what you reserved, not what you eat When a party calls ahead for four, the host holds a four-top. It doesn't matter if two of them only order drinks. The table is held for four. Pod requests work the same way. A container that requests 1 CPU and 2 GiB of memory takes 1 CPU and 2 GiB off the node's books when it's placed, even if it idles at 50 millicores. The scheduler never looks at actual usage. It adds up the requests of every pod already on a node and compares the total with the node's allocatable capacity, which is a little less than the machine's size because the kubelet and the operating system keep some back. That's how a cluster can show 30% CPU usage on the dashboard and still leave pods Pending with Insufficient cpu. Every table is reserved. Nobody is eating much. Limits are a different thing. They're how much the party is allowed to order. Go over the CPU limit and the container is throttled. Go over the memory limit and it's killed (OOMKilled) and restarted. Limits play no part in seating. Reserved for four, one person eating. The scheduler only sees the reservation. Filter, then score, then seat A good host doesn't pick the first empty table. They rule out the tables that can't work, then pick the best of what's left. The scheduler works in the same two phases. Filtering removes every node that can't run the pod: not enough unrequested CPU or memory, a taint the pod doesn't tolerate, a required affinity rule it fails, a volume it can't reach. Scoring ranks the nodes that survive, for example preferring ones that spread replicas apart or balance resource use. The pod is bound to the highest-scoring node, and only then does the kubelet on that node start the containers. If filtering leaves zero nodes, nothing gets scored. The pod stays Pending, and the scheduler records why it rejected each node: 0/6 nodes are available: 2 Insufficient memory, 3 node(s) had untolerated taint {dedicated: gpu}, 1 node(s) didn't match Pod's node affinity/selector. Read that like a host's notes: two tables too small, three reserved for someone else, one in a section you didn't ask for. Each count points to a different fix. Reserved tables: taints and tolerations Some tables have a little "Reserved" card on them. The host won't seat a walk-in there. A taint is that card on a node. kubectl taint nodes gpu-1 dedicated=gpu:NoSchedule tells the scheduler to keep pods off the node unless they carry a matching toleration. Control-plane nodes usually carry a taint like this, which is why your app pods never land on them. The part that confuses people: a toleration lets a pod sit at a reserved table, but it doesn't send it there. It's a pass, not a seat assignment. A pod that tolerates the GPU taint can still land on any ordinary node. To get GPU pods onto GPU nodes and nowhere else, you need both: the toleration (allowed in) and node affinity or a nodeSelector (sent there). The card comes in three kinds. NoSchedule means no new seating. PreferNoSchedule means seat here only if there's nowhere else. NoExecute also asks the people already sitting there to leave, so running pods without a matching toleration are evicted. A toleration is a pass to the reserved section. Affinity is the host walking you there. Seating requests: node affinity "Can we sit on the patio?" is a seating request. Hosts treat some requests as must-haves and some as nice-to-haves. Node affinity has the same two flavors. requiredDuringSchedulingIgnoredDuringExecution is a must-have: only nodes in zone us-east-1a, or only nodes labeled disktype=ssd. If no node matches, the pod waits. preferredDuringSchedulingIgnoredDuringExecution is a nice-to-have: the scheduler scores matching nodes higher but will seat you somewhere else. The long name tells you something useful. "IgnoredDuringExecution" means the rule is checked only at seating time. If someone relabels the node later, the pod keeps its table. Don't seat them together: anti-affinity and spread A good host doesn't seat one big office party across every table in a single section, and doesn't put a couple right next to the couple they just broke up with. Topology spread constraints handle the first case: spread this deployment's replicas evenly across zones or nodes, with a maxSkew saying how uneven it may get. Pod anti-affinity handles the second: don't put this pod on a node, or in a zone, that already runs a pod labeled app=checkout. This is how you keep three replicas from landing on one node, where a single hardware failure takes all three down. It's also a classic cause of Pending. Ask for required anti-affinity across nodes with five replicas and only four nodes, and replica five waits until a fifth node appears. The coat check is in the other room: volumes This one catches stateful workloads. Your coat is checked in the east dining room, so the host can only seat you in the east room. A persistent volume backed by a cloud disk lives in one availability zone, and a pod that mounts it can only run on nodes in that zone. If those nodes are full, the pod waits even when the other zones are empty, and the event says volume node affinity conflict. Setting the StorageClass volumeBindingMode to WaitForFirstConsumer helps with new volumes: the disk is created in whichever zone the pod is scheduled to, instead of a zone picked in advance. The VIP and the bumped reservation: priority and preemption A regular calls to say the mayor will be there in ten minutes. There's no free table. Some restaurants would ask a lower-priority party to move. Kubernetes does this when pods have a PriorityClass. If a high-priority pod can't be placed, the scheduler looks for a node where evicting lower-priority pods would make room. It evicts them, they get their normal graceful shutdown, and the high-priority pod is seated once the space frees up. The evicted pods go back to Pending and wait in line like everyone else. That's why cluster components such as DNS run at high priority, and why giving every workload the highest priority class defeats the point. Preemption: a higher-priority pod takes the table, and the lower-priority pod goes back on the waiting list. Opening the patio: the cluster autoscaler When the waiting list gets long enough, the manager opens the patio. The Cluster Autoscaler, or Karpenter, watches for pods that are Pending because no node fits them. When it finds one, it adds a node that would fit, and the scheduler seats the pod once the node joins. That takes minutes, not seconds, because a new machine has to boot and register. The patio only helps if a patio table would actually work for the party. Before scaling, the Cluster Autoscaler simulates whether the pod would fit on a new node from each node group, and Karpenter launches only capacity that meets the pod's constraints. A pod waiting on a taint nobody tolerates, an affinity rule no node group can match, or a volume in a zone with no capacity to add stays Pending no matter how long the line gets. The host never moves seated guests One last rule, and it surprises people. Once a party is seated, the host leaves them alone. If a better table opens up, nobody gets moved. The scheduler places a pod once and never revisits the decision. Add three empty nodes and the old ones stay crowded, because only new pods land on the new nodes. To rebalance, run the Descheduler, a separate project that evicts pods so the scheduler can place them again, or restart the deployment with kubectl rollout restart. Reading Pending like a host What the event says The restaurant version First thing to try Insufficient cpu or Insufficient memory Every table big enough is already reserved Set requests close to real usage, or add nodes had untolerated taint The only free tables are reserved Add the toleration, or use nodes without the taint didn't match Pod's node affinity/selector No free table in the section you insisted on Check node labels, or make the rule preferred didn't match pod anti-affinity rules or topology spread constraints You can't sit next to your own party Add nodes or zones, or soften the rule volume node affinity conflict Your coat is in the other room Add capacity in the volume's zone No scheduling events at all The host never saw you Check schedulerName and that the scheduler is running The takeaway Pending is never random. The scheduler checked every node against a short list of rules, and the event message tells you which rule turned you away from each one. Most of the time it's the reservation size: requests set far above what the app uses, filling tables nobody eats at. Once the pod is seated, its trip is only half done. I followed the rest of it in kubectl apply Said "created." 11 Steps Later, a Pod Gets Traffic. What's the strangest reason you've seen a pod stuck at the host stand?