Cilium and eBPF: Cluster Networking Without kube-proxy

The first time I ran iptables-save | wc -l on a Kubernetes node I got a number in the low thousands, on a cluster running about forty pods. That’s the honest scale of what kube-proxy does on your behalf. Every Service you create becomes a set of chains: one to match the ClusterIP, one per backend endpoint, a probability jump to spread load between them, a masquerade rule on the way out. It works, it has worked for a decade, and it is a linked list masquerading as a routing table.

Cilium’s pitch is to delete all of it. Run the cluster with no kube-proxy at all, and let eBPF programs attached to the kernel’s networking hooks do the service lookup out of a hash map instead of walking chains. I’ve been running this on a small bare-metal cluster for a while now, and the interesting part turned out to be a different thing entirely from the performance story everyone leads with. Let me get the mechanics down first, then say what actually changed day to day.

What kube-proxy actually does, and why it starts to hurt

A Kubernetes Service is a lie the node tells the pod. There is no process listening on the ClusterIP. When a pod connects to 10.96.0.1:443, something has to notice that packet and rewrite its destination to one of the real pod IPs behind the Service. In iptables mode, that something is a chain in the nat table, installed by kube-proxy, which watches the API server for Endpoints changes and rewrites the rules whenever they move.

The rules are evaluated linearly. For a cluster with a handful of services, this costs nothing measurable. The problem is the update path rather than the lookup path: when endpoints churn, kube-proxy recomputes and reloads a large slice of the ruleset, and iptables-restore takes a lock while it does. On a big cluster that turns into seconds of latency on every deployment. On a homelab cluster it turns into approximately nothing, and I want to be honest about that rather than sell you a performance problem you don’t have.

The second problem is legibility, and this one does bite at any size. When a Service doesn’t work, the diagnostic is reading generated iptables chains with names like KUBE-SEP-QW3JQ7XN6PVUBLZG and mentally simulating packet flow. I have spent evenings doing this. It’s the kind of debugging where you can be perfectly rigorous and still get the wrong answer, because the state you’re reading is a compiled artefact of a controller’s opinion, several rewrites removed from what you configured. Anyone who has done a full session of tcpdump and nsenter archaeology on container networking knows the feeling.

eBPF, in one paragraph you’ll actually believe

eBPF lets you load a small, verified program into the running kernel and attach it to a hook — a socket operation, a network device’s ingress path, a syscall. The verifier checks it terminates and can’t read memory it shouldn’t, then it’s JIT-compiled to native code. It gets kernel-speed execution with none of the “one bad pointer and the machine is gone” risk of a kernel module, and you can load and unload it live. That’s the whole trick, and I’ve gone into what it means for security surfaces in a longer piece on eBPF.

What Cilium does with it: service backends live in an eBPF hash map keyed by ClusterIP and port. When a pod calls connect(), a program attached at the socket layer rewrites the destination address before the packet is ever constructed. There’s no NAT on the wire, because the packet was born addressed to the right pod. Traffic between pods on the same node never touches the network stack’s routing layer at all — the program moves it directly between the two socket buffers.

Two things fall out of that which matter more than the throughput numbers. The lookup is a hash-map read, so it’s O(1) in the number of services. And because the rewrite happens at connect time, the pod’s connection tracking, the conntrack table entries, and what ss reports inside the pod all agree with each other. Under kube-proxy they don’t, which is why “the pod thinks it’s talking to the ClusterIP but tcpdump shows the pod IP” is a permanent source of confusion.

Installing it, and the one flag that matters

You need a kernel new enough to have the hooks — 5.10 is a sane floor, and anything shipping in 2024 is fine. You need the cluster to have no kube-proxy, which is easy on a fresh cluster and a small chore on an existing one. On K3s that means --disable-network-policy --flannel-backend=none --disable-kube-proxy at install time; on kubeadm it’s --skip-phases=addon/kube-proxy; on a cluster that already has it, delete the DaemonSet and clean the rules afterwards.

Cilium needs to know where the API server is, because with kube-proxy gone there’s no ClusterIP for it to bootstrap against. That’s the flag people miss:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

helm repo add cilium https://helm.cilium.io/
helm install cilium cilium/cilium --version 1.15.6 \
  --namespace kube-system \
  --set kubeProxyReplacement=true \
  --set k8sServiceHost=192.168.1.50 \
  --set k8sServicePort=6443 \
  --set routingMode=native \
  --set ipv4NativeRoutingCIDR=10.244.0.0/16 \
  --set autoDirectNodeRoutes=true \
  --set hubble.relay.enabled=true \
  --set hubble.ui.enabled=true

k8sServiceHost and k8sServicePort are the chicken-and-egg fix: Cilium talks to the API server directly at that address until it has installed the programs that would let it use a ClusterIP. Get this wrong and the agent pods sit in CrashLoopBackOff complaining they can’t reach the API server, which reads like a certificate problem and is a bootstrapping problem. If Helm’s abstractions make you twitchy, the flags map to a ConfigMap and a DaemonSet and you can read them directly — the reasoning in Helm charts demystified applies here as much as anywhere.

routingMode=native with autoDirectNodeRoutes=true is the setting I’d argue for on a homelab. It means no VXLAN or Geneve encapsulation: pod traffic goes onto the wire as pod IPs, and each node installs a route to every other node’s pod CIDR. It’s faster, the packets are readable in a capture, and your MTU stays at 1500 instead of losing 50 bytes to a tunnel header. The catch is that it only works when every node is on the same L2 segment, which in a flat home network is exactly the case. If your nodes are split across subnets, use the default tunnel mode and stop thinking about it.

Then verify:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

cilium status --wait

# prove the eBPF service map is real
kubectl -n kube-system exec ds/cilium -- \
  cilium-dbg service list

# ID   Frontend            Service Type   Backend
# 1    10.96.0.1:443       ClusterIP      1 => 192.168.1.50:6443 (active)
# 2    10.96.0.10:53       ClusterIP      1 => 10.244.0.14:53 (active)
#                                         2 => 10.244.1.9:53 (active)

That output is the entire service routing table of the cluster, in a form you can read without a decoder ring. Compare it to iptables-save and the appeal becomes hard to argue with.

Hubble is the bit that sells it

The performance case for eBPF is real and mostly irrelevant to a homelab. The observability case is what actually changed how I work.

Because Cilium already sits at every packet’s ingress and egress hook with a program that knows the Kubernetes identity of both ends, it can emit a flow record for every connection, labelled with the source and destination pod, namespace, and verdict — for free, without a sidecar, without instrumenting anything. Hubble is the interface to that.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

cilium hubble port-forward &

# every flow, live, labelled with real identities
hubble observe --follow

# only what is being dropped, which is the question you actually have
hubble observe --verdict DROPPED --follow

# what talks to my database, ever?
hubble observe --to-pod default/postgres-0 --last 500

The first time I ran hubble observe --verdict DROPPED on my own cluster it took about ninety seconds to find two things I’d been wrong about for months: a CronJob quietly failing to reach an internal endpoint, and a monitoring agent hammering a service that had moved. Neither had ever produced a log line anywhere. That’s the actual product. A network policy that drops traffic now tells you what it dropped and which policy did it, which turns network policy from a thing everyone plans to enable eventually into a thing you can enable on a Tuesday.

Network policies themselves become considerably more interesting too, because Cilium enforces on identity rather than IP, and understands L7 — a policy can say “the frontend may issue GET to /api/v1/orders on the backend, and nothing else”:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20

apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: frontend-to-backend
spec:
  endpointSelector:
    matchLabels:
      app: backend
  ingress:
    - fromEndpoints:
        - matchLabels:
            app: frontend
      toPorts:
        - ports:
            - port: "8080"
              protocol: TCP
          rules:
            http:
              - method: "GET"
                path: "/api/v1/orders.*"

That’s a service-mesh feature without a service mesh, no sidecar, no mTLS certificate rotation to babysit.

The identity model underneath is worth understanding, because it explains why the whole thing behaves the way it does. Cilium assigns a numeric security identity to each unique set of pod labels, and that identity travels with the packet — in native routing mode, in a field of the IP header; in tunnel mode, in the encapsulation header. Policy is evaluated against identities rather than addresses. The practical consequence is that a pod being rescheduled onto a different node with a different IP changes nothing at all: same labels, same identity, same policy verdict. Under an IP-based policy engine that reschedule is a window during which the rules and reality disagree, and you get exactly the sort of intermittent failure nobody can reproduce. Here the address is an implementation detail that policy never looks at.

It also explains Hubble’s labelling. The agent doesn’t infer which pod a packet belongs to by cross-referencing IPs against the API server after the fact — the identity is in the packet, so the flow record is correct by construction rather than by lookup. That’s why it can label flows at line rate without any of the sampling and guesswork a traditional flow collector needs. cilium-dbg identity list will show you the whole table, and it’s a short read: identities are per label-set rather than per pod, so a Deployment with twelve replicas is one identity.

What actually goes wrong

Agents crash-looping at install. Nine times in ten it’s k8sServiceHost pointing somewhere wrong, or kube-proxy still running and fighting for the same rules. kubectl -n kube-system get ds kube-proxy and make sure it’s gone.

Stale iptables rules after removing kube-proxy. Deleting the DaemonSet leaves its rules in place, and they keep intercepting traffic while Cilium’s programs also try to handle it. The result is a cluster that half works, which is worse than one that doesn’t. Run kube-proxy --cleanup on each node, or iptables-save | grep -v KUBE | iptables-restore, and reboot if you value your evening.

DNS resolves for some pods and not others. Almost always MTU. If you set routingMode=native on a network that’s actually doing encapsulation somewhere, large replies fragment and vanish while small ones sail through — so ping works and DNS is fine until a reply crosses the threshold. cilium status reports the MTU it detected. This failure mode is a close cousin of the ones in what actually happens when a pod looks up a name, and it costs everyone an afternoon exactly once.

LoadBalancer services stay Pending. Cilium replaced kube-proxy, and it did not become a bare-metal load balancer by doing so. You still need something to answer ARP for a virtual IP — either MetalLB alongside, or Cilium’s own L2 announcements feature, which does the same job with one fewer component. Either is fine; running both is a fight over who owns the address, and I’ve laid out the general shape of this in MetalLB and bare-metal load balancers.

cilium-dbg is your friend. cilium-dbg endpoint list shows every pod’s identity and policy state; cilium-dbg monitor is a firehose of raw datapath events. When Hubble’s opinion and reality disagree, monitor is the ground truth.

Memory. Each agent wants somewhere around 200–300MB steady-state, plus Hubble Relay and the UI if you enable them. On a Raspberry Pi cluster that’s a meaningful fraction of the machine, and it’s the honest counterargument to running this on the smallest possible hardware. On mini PCs with 16GB it’s noise.

The honest verdict

If you’re running Kubernetes at home to learn how Kubernetes works, Cilium is worth it for Hubble alone. Ignore the throughput pitch — you will not measure the difference on a gigabit LAN with forty pods, and anyone telling you otherwise is quoting a benchmark from a cluster a hundred times the size of yours. The payoff is that it makes the network legible. Fifteen years of “the packet went somewhere and I don’t know where” becomes a labelled flow log you can filter by pod name. That’s the upgrade.

If you’re running Kubernetes at home because you want the services on it to stay up with minimum fuss, Flannel is 40MB of memory and never thinks about anything, and there is no shame in that. The complexity Cilium adds is real: more moving parts, a genuinely deep configuration surface, and failure modes that require you to understand eBPF at least dimly to diagnose. I’ve laid out that fork in the road more fully in Calico vs Cilium for a home cluster, and my answer hasn’t changed: pick Cilium when you want to see, pick something dumber when you want to forget.

What tipped me was noticing I’d stopped guessing. Every network question I had about my own cluster — who talks to what, what’s being dropped, why this thing can’t reach that thing — became a command with an answer instead of a hypothesis with a tcpdump. Deleting kube-proxy was almost incidental. The thing worth having is knowing what your cluster is doing.