Load Balancing Is Just a Bouncer With a Clipboard

I spent a long time assuming load balancers did something clever with packets. They mostly do something clever with a list. A load balancer sits in front of a venue with several identical rooms behind it, watches who’s already inside each one, and decides which door the next person goes through. The clipboard is the state it keeps: which backend is up, how busy each one is, and which algorithm decides who queues where. Once you see it that way, the terminology stops being intimidating and starts being obvious.

The reason this matters at home as much as at a company running thousands of servers is that the same primitive shows up everywhere you’d never expect it. A reverse proxy in front of two Docker containers running the same app is a load balancer. keepalived moving a floating IP between two boxes is a small load balancer. Even DNS round robin, badly, is trying to do the bouncer’s job without a clipboard, which is exactly why it handles failure so poorly — a DNS record has no idea whether the IP it’s handing out is actually alive.

What the clipboard actually records

A load balancer needs three pieces of information before it can make a decision on a single request: which backends exist, whether each one is currently healthy, and which algorithm to apply when more than one healthy backend is available. Strip away the marketing vocabulary and every load balancing product, from a five-line nginx config to a large cloud provider’s managed balancer, is answering those same three questions.

The algorithm choice is where people spend the most time bikeshedding, and it matters less than the health check. Round robin sends request one to backend A, request two to backend B, request three back to A, forever, regardless of how loaded either one is. It’s the college-bouncer version of load balancing: strictly alternating doors, no judgement about who’s already inside. Least connections is smarter — it sends the next request to whichever backend currently has the fewest open connections, which matters a lot if your requests vary wildly in how long they take to answer. Weighted round robin adds a multiplier per backend, so a beefier server in a mixed fleet gets, say, three requests for every one a smaller box gets — useful the day you add a faster machine to a pool of older ones and don’t want to retire the old ones outright. IP hash pins a given client IP to the same backend every time, which matters if your app keeps session state in memory rather than in a shared store.

None of these algorithms save you from a dead backend on their own. That’s the health check’s job, and it’s the part people skip when they’re in a hurry.

Health checks: the ID check at the door

A bouncer with a clipboard but no eyesight is useless. If they can’t tell whether a room is actually open, they’ll keep sending people to a locked door. That’s what a load balancer without health checks does: it keeps routing traffic to a backend that crashed three minutes ago, because nothing told it to stop.

A health check is a small, repeated probe — hit /healthz every few seconds, expect a 200, mark the backend down after a handful of consecutive failures, mark it back up after a handful of consecutive successes. The exact numbers matter more than people expect. Too aggressive (mark down after one failure) and a single slow garbage collection pause takes a perfectly healthy backend out of rotation. Too lax (mark down after twenty failures) and users hit a dead backend for the better part of a minute before the load balancer notices.

The distinction between a liveness check and a readiness check is worth being precise about, because Kubernetes borrowed this vocabulary and a lot of people use it without knowing why there are two. Liveness answers “is the process still running.” Readiness answers “is the process currently able to serve traffic.” A service that’s still alive but is blocked reconnecting to its database is live and not ready — you want the load balancer to stop sending it requests without restarting the process. Conflating the two is one of the most common reasons a rolling deploy causes a brief spike in errors: the load balancer kept sending traffic to a pod that had started but hadn’t finished warming up.

Sticky sessions: when you deliberately break the fairness

Sometimes you want the bouncer to remember a specific person and always send them through the same door, even if that door has a longer queue. That’s a sticky session — usually implemented with a cookie the load balancer sets on the first response, then reads on every subsequent request to route that client back to the same backend.

The honest reason most homelabbers end up needing this is session state stored in a backend’s own memory rather than in a shared store like Redis or Postgres. If a login session lives only in the process that issued it, and the next request lands on a different backend that’s never heard of that session, the user gets logged out at random. Sticky sessions paper over this, but they don’t fix the underlying fragility — the moment that one backend dies, every session pinned to it is gone, and you’re back to the exact problem load balancing was meant to solve. The more durable fix is moving session state out of the application process entirely, which is one of the strongest arguments for treating Postgres as a cache you already understand rather than reaching for a separate Redis instance in a small setup — one fewer moving part to keep sessions consistent across.

Layer 4 versus layer 7: how deep the bouncer looks

This is the distinction that actually changes what a load balancer can do for you, and it’s worth being concrete about it rather than reciting the OSI model.

A layer 4 (transport-layer) load balancer looks at IP addresses and TCP/UDP ports and nothing else. It forwards packets without understanding what’s inside them. This is fast and cheap, and it’s what you want for raw TCP services, databases behind a balancer, or anything where you don’t want the extra hop of terminating and re-establishing a connection. keepalived operating with a virtual IP, or a bare iptables DNAT rule doing round robin, is layer 4 balancing — I’ve covered building high availability with a floating IP and keepalived before, and it’s the simplest possible version of this pattern, with no per-request decision-making at all.

A layer 7 (application-layer) load balancer terminates the connection, actually reads the HTTP request, and can make routing decisions based on the path, the hostname, cookies, or headers. This is what lets you run app.example.com/api and app.example.com/static on entirely different backend fleets behind one address, or route based on a canary cookie for a gradual rollout. It costs you an extra TLS termination and a bit of latency, and it buys you almost everything people actually want a load balancer for day to day. Caddy and Traefik both operate at this layer, and if you’re choosing between them for a home setup, the honest trade-offs are laid out in Caddy versus Traefik as reverse proxies.

Building the smallest useful version yourself

You don’t need a managed cloud load balancer to see this in action. HAProxy has been doing this job, unglamorously and reliably, since before “cloud native” was a phrase anyone used. Here’s a config that runs two backend containers behind one address with active health checking:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

frontend web
    bind *:80
    default_backend app_servers

backend app_servers
    balance leastconn
    option httpchk GET /healthz
    http-check expect status 200
    server app1 10.0.0.11:8080 check inter 3s fall 3 rise 2
    server app2 10.0.0.12:8080 check inter 3s fall 3 rise 2

Read that config line by line and it maps directly onto the three questions from earlier. balance leastconn is the algorithm. server app1 ... check inter 3s fall 3 rise 2 is the health check, probing every three seconds, marking down after three consecutive failures, marking back up after two consecutive successes. The backend list — app1 and app2 — is the clipboard. Nothing here is more mysterious than that.

If you’re already running everything through a reverse proxy, you may not need a separate load balancer tier at all. Caddy and Traefik will both distribute across multiple upstreams for a single service definition, health-check them, and terminate TLS in the same process, which is usually enough for a home setup with two or three replicas of the same app. Where a home lab actually benefits from a dedicated layer 4 balancer is Kubernetes on bare metal, where MetalLB fills the gap that a cloud provider’s load balancer would normally fill — I’ve written about getting real load balancer IPs on a bare-metal Kubernetes cluster with MetalLB if you’re running a cluster rather than a handful of containers.

Connection draining: closing a door without shoving people out

Taking a backend out of rotation sounds simple until you realise some of the people already inside haven’t finished. Connection draining is the load balancer’s way of closing one door gracefully: stop sending new requests to a backend marked for removal, but let existing connections finish naturally instead of cutting them off mid-response. HAProxy calls this a graceful shutdown; Kubernetes achieves the same thing with a preStop hook and a termination grace period long enough for in-flight requests to complete.

Skipping this is one of the more embarrassing ways to cause an outage you didn’t need to cause. A rolling restart without draining looks fine in a health check dashboard — the new backend comes up, the old one goes down — while a handful of real users watch their in-progress upload or long-running API call get severed halfway through. The fix costs nothing except patience: set the grace period a few seconds longer than your slowest realistic request, and let the old backend finish what it started before it disappears from the clipboard.

Troubleshooting: when the bouncer gets it wrong

The single most common failure mode is uneven load with a “working” health check. If one backend is consistently getting three times the traffic of another with round robin balancing, check whether the client is reusing a single long-lived connection — HTTP keep-alive means one client can pin an entire session’s worth of requests to whichever backend answered the first one, which looks like broken balancing but is actually connection reuse working exactly as designed.

The second most common failure is a health check that passes but a backend that’s still failing real requests. This usually means the health check endpoint is too shallow — it returns 200 as long as the process is running, without checking that the database connection pool, the cache, or any dependency the app actually needs is reachable. A useful health check does a small amount of real work: query the database with a trivial SELECT 1, check that the cache connection is live, and only then return success. It costs a few milliseconds per check and it’s the difference between “healthy” meaning something and “healthy” meaning “hasn’t crashed yet.”

The third is flapping: a backend oscillating between marked-up and marked-down every few seconds. This is nearly always the fall/rise thresholds set too aggressively for a backend with genuinely variable response times under load. Widening the window (more consecutive checks required before a state change) trades a few seconds of detection latency for a lot less oscillation, and it’s almost always worth the trade.

A fourth, sneakier one shows up only after a deploy: every request briefly fails during a rolling restart even though health checks pass instantly. This is usually a race between the container reporting itself ready and the application actually finishing its startup work — connection pool warmup, cache priming, or a migration check. Adding a short deliberate delay before the health check starts passing, rather than trusting the process’s own idea of “ready,” closes the gap.

Is it worth setting up at home

If you’re running a single instance of everything, no. A load balancer in front of one backend is a proxy with an unused clipboard. It earns its keep the moment you have two or more replicas of anything you care about staying up during a deploy or a crash — and at that point it’s not optional so much as the thing that turns “the site is down while I restart the container” into “nobody notices.” Start with whatever reverse proxy you already run doing the balancing; reach for HAProxy or MetalLB only once you’ve outgrown what Caddy or Traefik can express in a single config block, and reach for Redis-backed session storage before you reach for sticky sessions as the permanent fix.