Kubernetes Is a Very Expensive `while true` Loop

Somewhere around my third production Kubernetes incident, at 2am, staring at a Deployment that kept recreating a crashing pod every few seconds, it finally clicked what I was actually watching: a while true loop, running forever, comparing what exists to what should exist, and nudging reality one step closer every time it noticed a gap. That’s not a simplification I made up to feel clever about it afterwards — it’s the literal architecture. Every controller in Kubernetes, from the humble ReplicaSet controller to the most elaborate custom operator, is the same loop with different comparison logic bolted on.

That reframing matters because Kubernetes has a reputation, deserved in places, for being an intimidating pile of YAML and jargon — CRDs, admission webhooks, finalizers, reconciliation, the whole vocabulary. Underneath all of it sits one idea simple enough to write in fifteen lines of pseudocode, and once you actually hold that idea, the jargon stops being mysterious and starts being “oh, that’s just the loop, but for this particular resource.”

The loop, in its simplest possible form

Every Kubernetes controller follows the identical shape:

1
2
3
4
5
6

while True:
    desired = read_desired_state()   # what the YAML says should exist
    actual = read_actual_state()     # what's actually running right now
    if actual != desired:
        reconcile(actual, desired)   # take one step to close the gap
    sleep(requeue_interval)

That’s the entire architecture of a Deployment controller, a Service controller, a StatefulSet controller and every custom operator anyone has ever written for Kubernetes. The Deployment object you write in YAML is the desired half of the loop; the actual Pods, ReplicaSets and their statuses are the actual half; and the controller’s entire job is noticing a difference and taking one corrective step, then checking again. It never assumes the correction worked — it just loops back around and re-measures, which is precisely why a crashed pod gets recreated repeatedly rather than once: the loop doesn’t remember it already tried, it just sees a gap and closes it, forever.

Where this shows up concretely

Watch kubectl get events --watch on a cluster while you delete a pod that belongs to a Deployment, and the loop is visible in real time:

1
2
3

Pod default/web-7d9f8c6b5-x2k4p deleted
ReplicaSet default/web-7d9f8c6b5 noticed 2 replicas, wants 3
Pod default/web-7d9f8c6b5-n8j2q created

Nobody told the ReplicaSet controller “a pod just died, please replace it.” It simply woke up — either on a timer or because it was notified of a change via the API server’s watch mechanism — counted the actual pods, compared that count to spec.replicas, saw 2 instead of 3, and created one more. Kill that new pod immediately and the exact same sequence repeats, because the loop has no memory of having already fixed this once; it only knows the current gap between desired and actual.

This is also why kubectl apply feels different from kubectl create: apply doesn’t imperatively tell Kubernetes what to do, it just updates the desired-state object and lets the relevant loop notice the change on its own schedule and reconcile toward it. That’s the entire declarative model in one sentence — you’re never telling Kubernetes to do something, you’re only ever changing what “done” looks like and trusting the loop to get there.

Writing your own controller makes the pattern impossible to unsee

The clearest way to internalise this is to write a tiny custom controller yourself. A minimal one, watching a ConfigMap and reconciling based on its contents, is genuinely only a few dozen lines with client-go or a framework like kubebuilder:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

func (r *Reconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    var desired corev1.ConfigMap
    if err := r.Get(ctx, req.NamespacedName, &desired); err != nil {
        return ctrl.Result{}, client.IgnoreNotFound(err)
    }
    // compare desired.Data against whatever "actual" means for this resource,
    // take one corrective action if they differ, then return
    if needsAction(desired) {
        applyOneStep(desired)
    }
    return ctrl.Result{RequeueAfter: 30 * time.Second}, nil
}

RequeueAfter is the sleep(requeue_interval) from the pseudocode above, spelled out explicitly. Every operator you’ve ever installed from an ArgoCD dashboard or a Helm chart — cert-manager renewing certificates, an ingress controller reconciling load balancer rules, a database operator managing failover — is this exact function signature with different comparison and correction logic inside it. There is no special controller magic reserved for the built-in resources; custom controllers use the identical mechanism the core scheduler and kubelet use.

Even kubelet itself, the agent running on every node, is the loop pushed down one more layer: its desired state comes from the pods the API server says should be scheduled onto that particular node, its actual state is whatever containers the local container runtime reports running, and its correction step is starting or stopping containers to close the gap. Nest that same idea inside kubectl get nodes, inside the container runtime’s own reconciliation against the images it’s supposed to have pulled, and inside cgroups enforcing the resource limits attached to each container, and the entire Kubernetes stack turns out to be loops reconciling other loops’ outputs, all the way down to the kernel primitives underneath any single container.

Level-triggered, not edge-triggered — the design choice that makes it resilient

There’s one detail in that pseudocode worth dwelling on, because it’s the actual source of Kubernetes’ resilience rather than an implementation detail: the loop is level-triggered, not edge-triggered. An edge-triggered system reacts to an event — “a pod just died, do something” — and if that reaction gets missed or fails, nothing retries until the next distinct event happens. A level-triggered system, which is what every Kubernetes controller actually is, ignores the history of events entirely and only ever asks “given the current state of the world, right now, is there a gap?” It doesn’t matter whether a pod died from an OOM kill, a node failure, a kubectl delete, or gremlins — the controller doesn’t care what caused the gap, only that one currently exists, and it’ll keep closing that gap on every pass regardless of how it got there.

This is why restarting a controller mid-incident is almost always safe, and often the correct move: a fresh controller process reads the current actual and desired state cold, with zero memory of what happened before it started, and immediately resumes reconciling correctly. An edge-triggered system restarted mid-incident can permanently miss the event that would have told it to act. Kubernetes controllers can’t miss anything that still matters, because the next loop iteration re-derives the entire picture from scratch every single time.

The scheduler is the same loop wearing a bin-packing hat

The kube-scheduler looks like a special, separate piece of machinery — arguably the most algorithmically interesting part of Kubernetes, deciding which node a pod should land on based on resource requests, taints, tolerations, affinity rules and a scoring function. Strip away the scoring logic and it’s the identical loop: desired state is “this pod exists and needs a node,” actual state is “this pod has no node assigned yet,” and the single corrective action available is binding the pod to whichever node scores highest. Watch it with:

1

kubectl get pods -o wide --watch

A pod sits in Pending for exactly as long as the scheduler’s loop hasn’t yet found a node satisfying its resource requests and limits — the loop keeps re-evaluating candidate nodes every pass until one clears the bar, or until you intervene by freeing capacity elsewhere. There’s no separate “scheduling engine” conceptually distinct from every other controller; it’s the same reconciliation pattern with a bin-packing decision as its correction step.

Why “expensive” is the right word, not an insult

Calling it an expensive loop isn’t a swipe at Kubernetes’ design — reconciliation loops are a genuinely good idea, more robust than imperative “do this, then this, then this” automation, because a loop that keeps re-checking recovers automatically from problems an imperative script would just crash on. The expense is real, though, and it’s worth being honest about where it comes from: running the loop’s infrastructure — the API server, etcd, the scheduler, the controller manager, kubelet on every node — costs meaningful CPU and memory before your actual workload runs a single request. Why a Kubernetes cluster crashes at 2am is very often the answer to “what happens when the loop’s own infrastructure runs short of the resources it needs to keep looping.”

For a single application on a single box, that overhead buys you nothing you couldn’t get from a systemd unit with Restart=always, which is itself a much smaller, much cheaper version of the identical reconciliation idea — check if the process is running, if not, start it, repeat. The loop pattern scales down to a five-line systemd config and scales up to a global fleet of thousands of nodes; Kubernetes is the version built for the fleet, and paying its overhead for a single box is buying infrastructure sized for a problem you don’t have.

Troubleshooting: reading incidents through the reconciliation lens

• A Deployment keeps recreating a pod that immediately crashes, forever. This is the loop working exactly as designed against a desired state that’s wrong — the fix is the underlying image or config, not the loop. kubectl describe pod and kubectl logs --previous on the crashed instance show why the actual state can never match what’s desired.
• A change to a ConfigMap doesn’t seem to take effect. Many controllers only reconcile on their own schedule or on a watch event tied to the specific object type they own — a ConfigMap change doesn’t automatically trigger a Deployment’s pods to restart unless something is explicitly watching for it (a checksum annotation, a sidecar like Reloader). The loop reconciled the ConfigMap itself correctly; nothing told the Deployment’s loop that its desired state changed too.
• kubectl apply appears to hang or do nothing. Check kubectl get events for admission webhook rejections or validation errors — the desired state was never actually accepted, so there’s nothing new for any loop to reconcile toward.
• A custom operator seems to reconcile constantly even when nothing changed. Look for RequeueAfter set too aggressively, or a reconcile function that always reports a difference because it’s comparing fields that naturally drift (timestamps, generated names) rather than the fields that actually matter.
• Two controllers appear to be fighting, flipping a resource back and forth. This happens when two loops both believe they own the same field of the same object’s desired state — a Helm-managed Deployment and a separate autoscaler both trying to set replicas, for instance. kubectl get <resource> -o yaml and checking managedFields shows which controller last wrote which field, and the fix is almost always narrowing which loop owns that specific field, not disabling either controller outright.
• A resource is stuck in a Terminating state and won’t go away. A finalizer registered by some controller is blocking deletion until that controller finishes its own cleanup reconciliation — the loop responsible for removing the finalizer hasn’t run successfully yet. kubectl get <resource> -o jsonpath='{.metadata.finalizers}' shows what’s still pending, and the fix is almost always fixing whatever’s preventing that controller’s loop from completing, rather than force-removing the finalizer and leaving orphaned external resources behind.

What the loop costs you in practice, beyond the control plane

The overhead spans the API server and etcd sitting idle waiting for something to reconcile, every controller’s watch connection, every informer cache mirroring cluster state in memory, and every reconcile pass that runs even when nothing actually changed, purely because the requeue timer fired. On a large cluster this is genuinely negligible against the workload it manages. On a three-node home cluster running a handful of self-hosted services, the control plane’s own resource footprint is frequently a meaningful fraction of the total capacity available, which is the specific complaint behind choosing Proxmox over Kubernetes for a home lab’s actual mess of workloads — you’re paying the loop’s fleet-scale tax on a fleet of three machines.

Is it worth reaching for

The reconciliation loop is a genuinely elegant idea, and once you see it, every part of Kubernetes stops being a separate thing to memorise and becomes the same loop with new nouns. That clarity doesn’t answer whether you need Kubernetes at all — plenty of workloads are perfectly served by a single systemd unit doing the same job at a tenth of the operational cost, and Proxmox VMs for the messier cases Kubernetes handles poorly are frequently the better fit for a home lab. What the loop model does give you is the ability to debug Kubernetes from first principles instead of memorising incantations — every incident is ultimately a question of what the desired state says, what the actual state is, and why the gap between them isn’t closing.