K3s Multi-Node: Adding a Second Machine to Your Cluster

A single-node K3s install is a wonderful thing. It’s Kubernetes that fits in a few hundred megabytes of RAM, it boots in seconds, and it lets you learn the whole ecosystem on one spare machine. But a cluster of one is a contradiction in terms. The day a sensible person finally gives in and orders a second mini PC — or rescues an old laptop from the cupboard — is the day K3s starts being genuinely useful. Pods can be rescheduled, you can drain a node to do maintenance, and a single dead disk stops being the end of the world.

Going from one node to two is delightfully undramatic. K3s was built for exactly this. I’ve done it on hardware ranging from a Raspberry Pi 4 to a fanless N100 mini PC, and the procedure is identical every time. Here’s how it actually works, and the handful of things that bite people — because the join itself takes two minutes, and the debugging afterwards is where the afternoon goes if you skip the boring parts.

The two roles: server and agent

K3s nodes come in two flavours. A server runs the control plane — the API server, scheduler, controller manager and the embedded datastore. An agent runs nothing but the kubelet and your workloads; it phones home to a server for instructions. Your existing single node is already a server. The simplest multi-node setup is to keep that server and bolt on an agent.

This asymmetry is the whole reason K3s is pleasant to grow. A full upstream Kubernetes cluster makes you stand up etcd, generate a pile of certificates, and wire the components together by hand. K3s bundles all of that into one binary and one systemd service, and joining a worker is a single install command with two variables set. You are not assembling a control plane; you are pointing a new kubelet at an existing one.

To join, an agent needs two things: the server’s URL and the cluster’s node token. The token lives on the server here:

1

sudo cat /var/lib/rancher/k3s/server/node-token

Copy that value. It’s a long string prefixed with K10 followed by a CA hash and a secret — that structure lets the agent both authenticate and verify it’s talking to the right control plane, so treat it like a password. While you’re on the server, note its LAN IP — say 192.168.1.10. That’s the address the agent will connect to, so it must be a stable one. Static IP or a DHCP reservation; do not skip this. If that address ever changes, every agent you’ve joined silently loses its control plane and you’ll be debugging a “cluster” of orphaned nodes.

Joining the second machine

On the new machine, the install is a single command with two environment variables:

1
2
3
4

curl -sfL https://get.k3s.io | \
  K3S_URL=https://192.168.1.10:6443 \
  K3S_TOKEN="<the-node-token-you-copied>" \
  sh -

Because K3S_URL is set, the installer knows this is an agent, not a new server, and configures it to register against your existing control plane. That single environment variable is the whole switch — omit it and you’d get a second, standalone server that knows nothing about the first. Give it thirty seconds, then check from the server:

1
2
3
4

$ kubectl get nodes -o wide
NAME      STATUS   ROLES                  AGE   VERSION
node01    Ready    control-plane,master   42d   v1.30.4+k3s1
node02    Ready    <none>                 38s   v1.30.4+k3s1

That <none> under ROLES is correct — it’s a worker. You now have a real cluster. The scheduler will start placing new pods across both nodes automatically. One thing worth doing immediately: keep the two nodes on the same K3s version. The installer grabs the current stable channel by default, so if your server was installed months ago you can end up with a newer agent than server, which K3s does not love. Pin both to a channel or a version (INSTALL_K3S_CHANNEL=v1.30 or INSTALL_K3S_VERSION=v1.30.4+k3s1) and upgrade them together.

The networking gotchas nobody warns you about

This is where the ten-minute job becomes a forty-minute job if you’re unlucky. K3s’s default Flannel CNI uses VXLAN over UDP port 8472, the agent talks to the API on 6443, and the kubelet metrics flow back on 10250. If you run a firewall — and you should — open those between your nodes:

1
2
3

sudo ufw allow from 192.168.1.0/24 to any port 6443 proto tcp
sudo ufw allow from 192.168.1.0/24 to any port 10250 proto tcp
sudo ufw allow from 192.168.1.0/24 to any port 8472 proto udp

Note that I’ve scoped these to the LAN subnet rather than opening them to the world. There is no reason for the wider internet to reach your kubelet, and 10250 in particular has been the entry point for real-world cluster compromises when left exposed. Lock it to the node network and move on.

Troubleshooting: the failures you’ll actually hit

Symptoms of getting the firewall wrong are maddeningly specific. The node shows Ready, but pods on different nodes can’t reach each other, or kubectl logs and kubectl exec hang forever. Nine times out of ten it’s the VXLAN port (8472/UDP) silently dropped — the control plane traffic on 6443 got through, so the node registers fine, but the pod network can’t form across hosts. Confirm it with a pod on each node:

1
2
3

# schedule a shell on node02, ping a pod's IP on node01
kubectl run probe --image=busybox --overrides='{"spec":{"nodeName":"node02"}}' \
  -it --rm -- ping -c3 <pod-ip-on-node01>

If that times out but same-node pods reach each other, it’s the VXLAN port every time.

The second classic trap is mismatched clocks. If the two machines’ times drift more than a few minutes apart, TLS handshakes fail with confusing certificate errors — “x509: certificate has expired or is not yet valid” on a cert that’s plainly fine. Run an NTP client (systemd-timesyncd is already there on most distros) on both and forget about it.

Third, watch for a node stuck in NotReady right after join. kubectl describe node node02 and journalctl -u k3s-agent -f on the agent are your two tools here. The usual culprits are a wrong or truncated token (copy-paste ate a character), a K3S_URL pointing at a hostname the agent can’t resolve, or the server firewall blocking 6443 inbound. Read the agent journal — K3s is unusually honest about what it’s failing to reach.

Labelling and steering workloads

Once you have two nodes, you’ll want some control over where things land. By default the scheduler spreads pods based on resource availability, which is usually fine, but a heterogeneous homelab often isn’t uniform — one node has the fast NVMe, the other has the eGPU, or one is a low-power ARM board you’d rather keep for light services. Labels are how you express that:

1
2
3
4
5

# tag the beefy node
kubectl label node node02 workload=heavy
# then in a deployment's pod spec:
#   nodeSelector:
#     workload: heavy

For softer preferences use affinity/podAntiAffinity rather than a hard nodeSelector — anti-affinity in particular is what stops two replicas of the same service landing on the same node and defeating the whole point of having two. And if a node is genuinely special-purpose (say it hosts an accelerator you don’t want random pods scheduled onto), a taint plus matching toleration is cleaner than labels: the node repels everything except workloads that explicitly tolerate it.

The other command you’ll use constantly is kubectl drain node02 --ignore-daemonsets. That cordons the node and evicts its pods so they reschedule elsewhere, which is exactly what you want before a reboot or a kernel update — the entire reason you added the second node in the first place. kubectl uncordon node02 when you’re done. Getting comfortable with drain/uncordon is what turns “a second machine” into “a cluster I can actually maintain without downtime.”

A note on storage

The moment you have two nodes, the default local-path storage provisioner becomes a quiet liability. It writes volumes to whichever node the pod first landed on. Reschedule that pod to the other node and the data simply isn’t there — the PVC is bound, the pod starts, and your database comes up empty or refuses to start on a directory that doesn’t exist. For stateless workloads this is fine. For anything with a database, you now need either node affinity to pin it down or proper distributed storage. That’s a bigger conversation — I’ve written up the trade-offs in Longhorn vs OpenEBS for Kubernetes storage, and if your data already lives on a NAS, using a QNAP as an iSCSI/NFS backend is another honest option. For now, just know that two nodes plus local-path plus a wandering database is a data-loss trap waiting to spring.

Want high availability? Not like this

Adding an agent gives you workload resilience: lose the agent and pods reschedule onto the server. But the server is still a single point of failure — kill it and the whole control plane goes dark, kubectl stops answering, and nothing gets rescheduled because the thing that does the rescheduling is the thing that died. True HA needs an odd number of servers (three is the magic number) with an embedded etcd quorum, started with the --cluster-init flag on the first and --server pointing back at it on the rest. You also want a stable virtual IP in front of the API so clients don’t care which server is currently answering — keepalived and a floating VIP is the standard homelab way to do that without paying for a cloud load balancer. That’s a worthwhile next step, but it’s a different job. For a homelab, one server and one or two agents is a perfectly honest setup.

The verdict

Adding a second node to K3s is one of the highest reward-to-effort tasks in self-hosting. For the cost of a cheap mini PC and ten minutes, you go from a fragile single box to something that survives a node reboot without taking your services down. If you’re still on that single node and starting from scratch, my K3s on a Raspberry Pi walkthrough covers the first install before you get here.

Just respect the firewall ports and the clock, keep the versions matched, keep stateful workloads on a leash until you’ve sorted real storage, and don’t mistake a worker node for high availability. Get those right and your little cluster will quietly carry on when one machine doesn’t — which is the entire point of running Kubernetes in the first place. It won’t survive the control-plane node dying, and it won’t balance load, but for “one of my boxes can reboot and nothing goes down,” two nodes is exactly enough. Add the third server and the VIP later, when a reboot of the control-plane node starts to hurt as much as a reboot of a worker used to — and by then you’ll have the drain/uncordon muscle memory to do it without a second thought.