Talos Linux: A Kubernetes Homelab With No SSH

The first time I installed Talos Linux I spent ten minutes looking for the login prompt. There isn’t one. The console shows a dashboard — service states, network addresses, a scrolling log — and that is the entire local interface. There is no getty, no bash, no apt, no systemd you can poke at, and emphatically no SSH daemon. If you want to know what the machine is doing, you ask it over a mutual-TLS API with a client called talosctl. If you want to change something, you send it a new machine configuration and it reconciles.

That sounds like an inconvenience dressed up as a feature, and for the first afternoon it genuinely is. Then you notice something. Every other node I’ve ever built accumulated silt: a debug package installed at 2am, a sysctl typed by hand and never written to a file, a firewall rule that exists on three of four machines. The node drifts away from its own documentation and nobody notices until a rebuild fails. Talos makes that class of mistake structurally impossible, because there is no mechanism through which drift can enter. The disk holds the OS as a read-only squashfs image, /var is the only writable partition that survives a reboot, and the machine’s entire personality lives in one YAML file you keep in git.

Why remove the shell at all

The argument for an immutable node is usually made in security terms, and that part is real. A Talos node has roughly a dozen binaries on it and none of them is a shell. The classic post-exploitation move — land in a container, break out, then use the host’s tooling to pivot — runs out of tools almost immediately. There is no curl to pull a second stage, no useradd, no package manager, no cron. The API is the only door, it speaks mutual TLS, and the client certificate lives in your talosconfig rather than on the machine. Compared to hardening a general-purpose distro until it behaves — which I’ve written about at length in hardening SSH beyond the basics — the attack surface reduction is a different order of magnitude.

The operational argument matters more to me, though. Kubernetes nodes should be cattle, and everyone says so, and then everyone builds pets anyway because Ubuntu makes it so easy to just fix the one thing by hand. Talos removes the temptation by removing the capability. A node is a machine config plus a disk. To rebuild it you PXE-boot it and apply the config. To upgrade it you point it at a new installer image and it swaps the boot partition, reboots, and comes back — and if it fails to come back it rolls onto the previous image. The rebuild path is the only path, so it’s the path that stays tested.

The third reason is narrower and worth saying out loud: Talos ships exactly the components Kubernetes needs, configured the way Kubernetes wants them, and nothing else. The kubelet, containerd, etcd, the CNI hooks, the kernel with the right modules compiled in. No snapd waking up mid-scrape, no unattended-upgrades restarting containerd on a Tuesday, no NetworkManager arguing with your CNI over an interface. The variance you spend a weekend eliminating on a general-purpose distro simply doesn’t exist here.

There’s a cost to all this, and I’d rather state it before the walkthrough than bury it in the verdict. You are trading familiarity for guarantees. Every debugging instinct you’ve built over fifteen years of Linux — strace the process, check journalctl, install the tool you need, read the config in /etc — has to be re-learned as a talosctl verb or a Kubernetes pod. For the first week that is a real productivity tax, and if your cluster is currently on fire, adopting Talos is a bad way to put it out. Do the migration when things are calm.

Building a cluster

You need an ISO or a raw disk image from the Talos releases page — as of the 1.7 series, which is what I’m running, it ships with Kubernetes 1.30. Boot the target machine from it and Talos comes up in maintenance mode: it grabs a DHCP address, prints it on the console dashboard, and waits for a config over the insecure API on port 50000. Nothing else works until you send one.

Generate the config on your workstation. Split the secrets out first, because you want those in a password manager rather than regenerated on a whim:

1
2
3
4
5
6
7
8

# secrets are the cluster's identity — generate once, keep them safe
talosctl gen secrets -o secrets.yaml

talosctl gen config mylab https://192.168.1.50:6443 \
  --with-secrets secrets.yaml \
  --output-dir _out

# _out/controlplane.yaml, _out/worker.yaml, _out/talosconfig

The endpoint URL in that command is the address of the Kubernetes API server as clients will reach it. On a single control-plane homelab that’s just the control-plane node’s IP. If you plan to add control planes later, put a virtual IP or a load balancer address there now — changing it afterwards means regenerating certificates, and you’ll be cross with yourself.

The generated controlplane.yaml is long and mostly sensible defaults. The parts you’ll actually edit are the install disk, the hostname and the network:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

machine:
  install:
    disk: /dev/sda
    image: ghcr.io/siderolabs/installer:v1.7.5
    wipe: false
  network:
    hostname: cp-1
    interfaces:
      - interface: eth0
        addresses:
          - 192.168.1.50/24
        routes:
          - network: 0.0.0.0/0
            gateway: 192.168.1.1
    nameservers:
      - 192.168.1.1
  time:
    servers:
      - pool.ntp.org
cluster:
  network:
    cni:
      name: none          # bring your own CNI
    podSubnets:
      - 10.244.0.0/16

Get the disk right. /dev/sda on a mini PC with one NVMe drive is usually /dev/nvme0n1, and Talos will happily tell you what it can see: talosctl -n 192.168.1.50 disks --insecure while the node is still in maintenance mode. Installing to the wrong disk is the single most common self-inflicted wound here, and on a machine with a USB stick still plugged in it’s a genuinely easy mistake.

Then apply, bootstrap, and pull a kubeconfig:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

talosctl apply-config --insecure \
  -n 192.168.1.50 -f _out/controlplane.yaml

export TALOSCONFIG=_out/talosconfig
talosctl config endpoint 192.168.1.50
talosctl config node 192.168.1.50

# bootstrap etcd — exactly ONCE, on exactly ONE control-plane node
talosctl bootstrap

talosctl kubeconfig ~/.kube/config
kubectl get nodes

The node reboots into the installed system, etcd forms, the control plane comes up, and kubectl get nodes shows a NotReady node because you set cni: none and nothing is providing pod networking yet. Install a CNI and it goes green. I default to Cilium on these clusters and have gone through that reasoning in Calico vs Cilium for a home cluster; Flannel works fine too, and Talos will install it for you if you leave the CNI default alone.

Workers are the same dance with worker.yaml and no bootstrap step. That’s the whole cluster build: two commands per machine, and the machine’s entire configuration is a file you can diff.

Living without a shell

This is the part that takes adjusting. Everything you’d normally reach for has a talosctl verb, and the vocabulary is small enough to learn in a week:

1
2
3
4
5
6
7
8

talosctl -n 192.168.1.50 dashboard        # live console, over the API
talosctl -n 192.168.1.50 services         # what's running and its health
talosctl -n 192.168.1.50 logs kubelet -f  # follow a service's logs
talosctl -n 192.168.1.50 dmesg -f         # kernel ring buffer
talosctl -n 192.168.1.50 get links        # network interfaces
talosctl -n 192.168.1.50 read /etc/resolv.conf
talosctl -n 192.168.1.50 containers -k    # kubernetes containers via CRI
talosctl -n 192.168.1.50 health           # cluster-wide readiness check

Note read, which fetches a file’s contents over the API. There is no write counterpart, and that’s deliberate — the machine config is how files get onto the machine. talosctl get speaks to Talos’s internal resource model, which is essentially the same controller-runtime idea Kubernetes uses, applied to the OS. talosctl get members shows cluster membership as the node itself understands it, which is a genuinely useful second opinion when kubectl is lying to you.

Changing configuration is a patch, applied over the API, versioned in git:

1
2
3
4
5
6

# patch a live node — no reboot needed for most fields
talosctl -n 192.168.1.50 patch machineconfig \
  --patch '[{"op":"add","path":"/machine/sysctls","value":{"vm.max_map_count":"262144"}}]'

# or edit interactively
talosctl -n 192.168.1.50 edit machineconfig

Talos tells you whether the change applies live or needs a reboot, and refuses politely if you’ve asked for something structurally impossible. Upgrades are one command and the node handles the rest:

1
2

talosctl -n 192.168.1.50 upgrade \
  --image ghcr.io/siderolabs/installer:v1.7.6

It cordons and drains itself, writes the new image to the inactive boot partition, reboots into it, and uncordons. If the new image fails to reach a healthy state, it falls back to the partition it came from. I have had this fire in anger once, on a node with a marginal SSD, and it did exactly what the documentation promised.

Extensions, and the day you need a driver

The obvious objection to an OS with no package manager is the day you need something that isn’t in the image. Intel GPU drivers for a transcoding node. iscsi-tools, because your storage layer wants an initiator on the host. util-linux-tools, because a CSI driver shells out to nsenter. On a normal distro that’s an apt install. On Talos it’s a system extension, which means the thing you install is a new boot image with the extension baked in.

Sidero’s Image Factory turns that into a web form and an API rather than a build pipeline. You pick a Talos version, tick the extensions, and it hands back a schematic ID — a hash of your choices — plus installer and ISO URLs pinned to it. The installer reference in your machine config then looks like this:

1
2
3
4
5

machine:
  install:
    disk: /dev/nvme0n1
    # schematic ID from the Image Factory, encoding the extension set
    image: factory.talos.dev/installer/<schematic-id>:v1.7.5

That URL is the whole story of what’s on the node. Somebody reading your git repo in two years can reconstruct the exact image bit for bit, which is more than I can say for any Ubuntu box I’ve ever built. Upgrading with extensions is the same talosctl upgrade command pointed at a new version of the same schematic.

The friction is real, though. Each extension change means a reboot, because the change is a new boot image. Iterating on “which driver do I actually need” is measured in reboots rather than seconds, and that is genuinely slower than the apt install loop. The upside is that once it works, it works on every node identically and forever, which is the trade Talos keeps offering you in different costumes.

The other storage question worth answering early: user disks. Talos will format and mount extra disks for you, declaratively, and it will refuse to let you do it by hand:

1
2
3
4
5

machine:
  disks:
    - device: /dev/sdb
      partitions:
        - mountpoint: /var/mnt/bulk

Anything under /var survives reboots and upgrades. Everything else is reconstructed from the image on every boot, which is exactly the property that makes drift impossible and makes people uncomfortable for the first fortnight.

What actually goes wrong

The node reboots into maintenance mode forever. Almost always a disk problem: the installer wrote to a USB stick, or the BIOS is booting the wrong device. talosctl -n <ip> disks --insecure in maintenance mode and check machine.install.disk against reality.

talosctl hangs with a certificate error after a rebuild. You regenerated secrets and the old talosconfig no longer matches. This is why gen secrets is a separate step and why the output belongs in your password manager. Regenerating cluster secrets means rebuilding every node.

bootstrap run twice, or on two nodes. You get a split etcd and a cluster that never converges. Bootstrap exactly once, on one control-plane node, ever. Recovery is wiping etcd and starting again, so read the command twice before pressing return.

Two control-plane nodes. Do one or do three. Two gives you a quorum of two, which means either node failing takes the cluster down — strictly worse availability than a single node, at double the power draw. This bites people who think of “add a second machine” as an availability win, which it is for workers and is a different calculation for control planes.

You genuinely need to poke the host. It happens — a driver question, a storage oddity. The escape hatch is a privileged debug pod with the host namespaces, which gets you a shell in a container that can see the host’s filesystem at /host:

1
2
3
4
5

kubectl run debug --rm -it --image=nicolaka/netshoot \
  --overrides='{"spec":{"hostNetwork":true,"hostPID":true,
    "containers":[{"name":"debug","image":"nicolaka/netshoot",
    "stdin":true,"tty":true,"securityContext":{"privileged":true},
    "command":["nsenter","--target","1","--mount","--uts","--ipc","--net","--pid","--","sh"]}]}}'

That works, and it feels like cheating, and it should feel like cheating. If you’re doing it weekly, Talos is fighting you.

Clock skew breaks everything at once. Talos is mutual-TLS end to end, and TLS is unforgiving about time. A node whose RTC battery has died and whose NTP servers are unreachable will fail certificate validation on every API call, and the error you get back reads like a certificate problem rather than a time problem. talosctl -n <ip> time compares the node’s clock to yours. On a segmented network where nodes can’t reach the internet, point machine.time.servers at your own gateway or NTP source and check that it’s actually answering.

talosctl health is not the same as kubectl get nodes. The former asks Talos whether etcd has quorum, whether the API server is serving, whether members agree on membership. The latter asks Kubernetes whether the kubelet has checked in recently. When they disagree, talosctl health is usually the one telling the truth earlier, because it can see a node that has stopped participating in etcd while its kubelet is still cheerfully reporting Ready.

Support bundles. When something is genuinely strange, talosctl support -O bundle.zip collects logs, resource state and service status from every node in one archive. It’s the first thing anyone will ask for, and it’s better than the ad-hoc rummaging you’d do on a normal box.

The honest verdict

Talos is the right choice when the machines exist to run Kubernetes and nothing else. If your control plane is three mini PCs that do one job, the immutability is pure profit: rebuilds are trivial, upgrades are boring, drift is impossible, and the security posture is excellent for free rather than for a weekend of hardening.

It’s the wrong choice when the box is doing five jobs. If the same machine runs Kubernetes and also holds a ZFS pool and also serves as your S3-compatible object store, you’ll spend your life fighting an OS designed to refuse exactly that kind of accretion. Talos supports user disks and extensions, and the extension system covers real needs — GPU drivers, iSCSI, util-linux tools — but the workflow is “rebuild the installer image with the extension baked in”, which is correct engineering and a poor fit for a machine you’re still experimenting with.

Backups deserve a sentence, because Talos changes the shape of them in a way that’s easy to miss. There is no OS state worth backing up — the OS is the image plus the config file, and both live in git. What you must back up is secrets.yaml, your talosconfig, and etcd. Talos does scheduled etcd snapshots to /var and talosctl etcd snapshot db.snap pulls one to your workstation on demand; that snapshot plus the secrets is a complete cluster restore. It’s a smaller and much better-defined backup surface than a general-purpose node, where you’re never quite sure what somebody put in /opt. It also means the usual advice applies with extra force: a snapshot you’ve never restored from is a hypothesis, which is the whole argument of testing your backups.

The learning curve is smaller than its reputation suggests, and it’s front-loaded. The first cluster takes an evening because you’re learning a new noun for everything. The second takes twenty minutes. The thing that surprised me was how much of the anxiety went away — I stopped being nervous about rebooting nodes, because a Talos node has nothing on it worth mourning. That’s the actual product. The missing shell is the mechanism; a cluster you’re willing to break is the result.

If you’re still on the fence, run it in a VM alongside the K3s cluster you already have and try to break it. Half an hour of talosctl and you’ll know whether the trade suits how you work.