Calico vs Cilium: Kubernetes CNI for a Home Cluster

Set up a fresh Kubernetes cluster and one of the first decisions you’re handed is also one of the least explained: which CNI? Your nodes sit there in NotReady, the docs wave vaguely at a list, and you pick whatever the tutorial used. Months later you’re trying to write a NetworkPolicy or debug a packet that vanishes, and you finally wish you’d understood the choice.

The Container Network Interface is the plugin that gives every pod an IP and makes pods on different nodes able to talk to each other. Without one, your cluster is a collection of isolated boxes. For self-hosters, two names dominate the conversation: Calico and Cilium. They reach the same destination by very different roads.

This decision sits right next to the other big homelab-cluster choices — the same trade-off between “boring and debuggable” and “powerful but complex” shows up in storage, where I weighed it in Longhorn vs OpenEBS for Kubernetes persistent storage, and it’s worth deciding the CNI deliberately rather than inheriting whatever a quick-start used. If you’re running k3s on a Raspberry Pi, this choice gets sharper still, because k3s ships its own defaults and overriding them has consequences.

What a CNI is on the hook for

Two jobs, fundamentally. First, connectivity: every pod gets an IP, and traffic finds its way across nodes, usually via an overlay network (encapsulating pod traffic inside node-to-node packets) or via plain routing where your network already knows the pod routes. Second, policy: NetworkPolicies that say “these pods may talk to those pods, and nothing else.” A bare cluster allows all pod-to-pod traffic by default, which is fine until it very much isn’t.

How a CNI does these two jobs is where Calico and Cilium part ways, and it comes down to one acronym.

Calico: routing you can reason about

Calico is the dependable, widely-deployed choice. Its standard mode does pod networking with straightforward Linux routing and iptables (or the newer eBPF dataplane if you opt in), and it can run as a pure layer-3 router that advertises pod routes over BGP — no overlay, no encapsulation overhead, packets that look like normal packets.

For a homelab the appeal is that it’s legible. When something breaks, you can ip route, iptables -L, and tcpdump your way to an answer using tools you already know. It’s been around a long time, the docs are excellent, and it does the job without drama. NetworkPolicy support is solid and it adds its own richer policy CRDs on top of the standard Kubernetes ones.

A minimal default-deny policy — the thing every cluster should have and almost none do — looks like this:

1
2
3
4
5
6
7
8
9

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-ingress
  namespace: prod
spec:
  podSelector: {}
  policyTypes:
    - Ingress

Apply that and every pod in prod rejects all incoming traffic until you write explicit allow rules. It’s the single highest-value security control most home clusters are missing, and Calico enforces it cleanly.

Installing it on a vanilla cluster is two manifests — the operator, then a configuration object that tells Calico how to assign pod IPs:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.28.0/manifests/tigera-operator.yaml
kubectl create -f - <<'EOF'
apiVersion: operator.tigera.io/v1
kind: Installation
metadata:
  name: default
spec:
  calicoNetwork:
    ipPools:
      - cidr: 10.244.0.0/16
        encapsulation: VXLANCrossSubnet
EOF

The VXLANCrossSubnet mode is the homelab sweet spot: it routes natively between nodes that share a subnet (no overhead) and only encapsulates when crossing subnet boundaries, which on a flat home network means almost never. If you want to drop encapsulation entirely and let Calico advertise pod routes to your router over BGP, that’s available too, but BGP at home is a “because I want to learn it” decision, not a “because I need it” one.

Cilium: eBPF and the programmable kernel

Cilium is the ambitious one. It’s built on eBPF, a technology that lets you run sandboxed programmes inside the Linux kernel without modules or patches. Instead of pushing traffic through long iptables chains, Cilium loads eBPF programmes that make forwarding and policy decisions in the kernel datapath directly. At scale that’s meaningfully faster, because iptables performance degrades as rules pile up and eBPF does not.

But Cilium is more than a faster dataplane. It does identity-based policy — rules keyed on pod identity rather than fragile IPs — and reaches up into layer 7, so you can write policies like “this pod may make GET requests to that API but not POST.” It ships Hubble, a genuinely lovely observability layer that shows live service-to-service traffic flows, which is the kind of thing you didn’t know you wanted until you’ve watched your cluster’s traffic light up in real time. It can even replace kube-proxy entirely.

The catch is complexity. eBPF requires a reasonably modern kernel, and when something does go wrong you’re debugging eBPF maps rather than iptables rules you can read. The power is real; so is the learning curve.

Cilium installs cleanly via its own CLI, and the flags you pick at install time are where you opt into the interesting parts:

1
2
3
4
5
6
7

cilium install --version 1.16.0 \
  --set kubeProxyReplacement=true \
  --set hubble.enabled=true \
  --set hubble.relay.enabled=true \
  --set hubble.ui.enabled=true
cilium status --wait
cilium hubble ui     # opens the live traffic map in your browser

That kubeProxyReplacement=true is doing real work: it removes kube-proxy from the cluster and handles service load-balancing in eBPF instead, which is both faster and one fewer moving part. Turning Hubble on is the bit you’ll actually enjoy — cilium hubble ui gives you a live service map showing exactly which pods are talking to which, with the verdict (allowed/denied) on every flow. The first time you apply a network policy and watch the denied flows light up red in real time, the eBPF complexity starts feeling worth it.

A layer-7 policy — the thing Calico can’t express in standard NetworkPolicy — looks like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20

apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: api-readonly
spec:
  endpointSelector:
    matchLabels:
      app: frontend
  egress:
    - toEndpoints:
        - matchLabels:
            app: backend
      toPorts:
        - ports:
            - port: "8080"
              protocol: TCP
          rules:
            http:
              - method: GET
                path: "/api/.*"

That permits the frontend to make GET requests to the backend’s /api/ paths and nothing else — no POST, no other path. Expressing application-layer intent in a network policy is genuinely powerful, and it’s the kind of thing that’s hard to give up once you’ve used it.

So which one for a home cluster?

Here’s my honest take after running both on hardware in my own house.

• If you want the path of least resistance, pick Calico. It’s stable, well-documented, debuggable with standard Linux tools, and it does everything a home cluster realistically needs. Most homelabs never push enough traffic for the eBPF performance gap to matter.
• If you’re here partly to learn, Cilium is a fantastic thing to learn on, and Hubble alone makes your cluster’s networking visible in a way nothing else does. The observability is the real prize at home scale, more than the raw performance.
• For policy, both enforce standard NetworkPolicy well. Cilium’s layer-7 and identity-based rules go further, but most people start and stop at “default-deny plus a few allows,” which either handles fine.
• For footprint, Calico is the lighter, simpler thing to keep running. Cilium asks a bit more of your kernel and your patience.

Troubleshooting: where these actually bite

Both CNIs have a small set of failure modes you’ll hit at home, and knowing them saves hours.

Nodes stuck NotReady, pods stuck ContainerCreating. This is almost always the CNI not being installed, or its pods crash-looping. Check kubectl get pods -n kube-system (or -n calico-system / Cilium’s namespace) and read the logs of whatever is restarting. A node only goes Ready once the CNI has set up its networking on that node.

Pod CIDR mismatch. If you told kubeadm --pod-network-cidr=10.244.0.0/16 but configured the CNI with a different range, pods get IPs that nothing routes. Make the two agree. With k3s this is the classic trap: k3s ships Flannel and its own kube-proxy by default, so installing Calico or Cilium on k3s means starting the server with --flannel-backend=none --disable-network-policy (and, for Cilium’s kube-proxy replacement, --disable-kube-proxy) or you end up with two CNIs fighting over the same pods.

Cilium needs a newer kernel than you have. The eBPF features that make Cilium worthwhile want a recent kernel; on an old distro you’ll see features silently disabled or the agent refusing to start. Check cilium status — it reports exactly which datapath features are active. If you’re on something ancient, this alone may decide the choice for you.

Your NetworkPolicy “doesn’t work.” Two usual causes. First, you applied a default-deny but forgot that DNS lives in kube-system — pods can no longer resolve names because you blocked egress to CoreDNS. Always pair a default-deny-egress with an explicit allow to the DNS service. Second, with Calico you’re using its CRDs but reading standard NetworkPolicy behaviour, or vice versa; they coexist but have different precedence rules. When in doubt, Cilium’s Hubble shows you the actual verdict on each flow, which turns “why is this blocked?” from guesswork into a glance.

MTU problems after enabling encapsulation. VXLAN adds overhead, and if your pod MTU isn’t reduced to account for it, large packets get fragmented or dropped and you get baffling “works for small requests, hangs on big ones” behaviour. Both CNIs usually auto-detect this, but on funny network setups you may need to set it explicitly.

The verdict

There’s no wrong answer here, which is the most reassuring thing I can tell you. If you want a CNI you install once and forget, that fades into the background and never surprises you, Calico is the safe and entirely respectable pick — it’s what I reach for when I just want the cluster to work. If you treat your homelab as a place to learn the modern, eBPF-shaped future of Linux networking, and you’ll actually use Hubble and the layer-7 policies, Cilium rewards the investment and is genuinely a joy once it clicks. Pick based on whether you want your CNI to disappear or to teach you something. Both are good. Just please, whichever you choose, write that default-deny policy.

One last practical note, because the choice isn’t quite symmetric in effort: swapping CNIs after the fact is painful. The cleanest path is to drain and rebuild, because two CNIs half-installed on the same cluster is a special kind of misery, and leftover iptables rules or eBPF maps from the old one cause exactly the sort of intermittent, hard-to-pin-down breakage you don’t want at home. So decide before you build, or at least be willing to flatten and start over when you change your mind. For most people standing up their first home cluster, that argues for Calico — get something stable and debuggable working, learn how pods and policies actually behave, and graduate to Cilium later if and when Hubble and layer-7 policy are things you’ll genuinely use rather than things that merely sound impressive.