Velero: Backing Up Kubernetes, Not Just Volumes
Cluster objects, persistent data, and a restore that actually recreates the cluster you had
Contents
The first time I lost a Kubernetes namespace to a fat-fingered kubectl delete namespace, I discovered that my “backups” covered exactly the wrong half of the problem. The persistent volumes were sitting safely on a ZFS pool with nightly snapshots. The Deployments, Services, Ingresses, ConfigMaps, Secrets and the dozen other objects that actually described how those volumes were supposed to be used were gone, and rebuilding them from memory and half-remembered YAML took the better part of an evening. A PersistentVolume without the Deployment that mounts it is just an orphaned disk; the cluster state is the application, and it needs backing up as deliberately as the data does.
That is the gap Velero closes. It backs up Kubernetes API objects and, through a snapshot integration, the persistent volumes those objects reference, as one coordinated operation, so a restore recreates both the data and the definitions that make sense of it.
Why volume backups alone are not enough
A homelab running k3s on a Raspberry Pi or a full multi-node cluster accumulates state in two very different places. The data lives in PersistentVolumes, which a filesystem-level backup tool can capture perfectly well. But the shape of the application — which container image and tag, how many replicas, what the Ingress routing looks like, which Secret a pod expects to find mounted at which path, what a HorizontalPodAutoscaler’s bounds are — lives entirely in the Kubernetes API, stored in etcd, and none of that is visible to a tool that only looks at disks. Losing a namespace, botching an upgrade, or needing to rebuild a cluster from scratch after a control-plane disk failure all destroy that API-level state even when every underlying volume is untouched. Velero backs up both halves together and, critically, restores them together, so a Deployment and the PersistentVolumeClaim it depends on come back in a consistent state rather than as two separately-restored pieces that may or may not still agree with each other.
Installing Velero
Velero needs a storage backend for the object metadata (an S3-compatible bucket works well for a homelab) and, for volume snapshots, either a cloud provider’s native snapshot API or a file-level plugin such as the built-in restic/kopia integration for clusters without cloud snapshot support — the normal case for a homelab running k3s on bare metal.
| |
Install the server component into the cluster, pointing it at an S3-compatible bucket and enabling the file-system backup (node-agent) integration for volume data:
| |
--use-node-agent deploys a DaemonSet that runs the file-system backup path (restic by default, kopia as a newer alternative) against pod volumes directly, which is what makes Velero viable on a bare-metal or homelab cluster with no cloud block-storage snapshot API underneath it. --default-volumes-to-fs-backup means every pod volume gets backed up this way without needing a per-pod annotation, the simplest default for a homelab where you’d rather over-back-up than miss something.
Taking and scheduling backups
A one-off backup of a whole namespace:
| |
That captures every object in the namespace plus the contents of any PersistentVolumeClaims it references. Watch it complete and inspect what it actually captured:
| |
For anything you care about, the real value comes from a schedule:
| |
This mirrors paperless-ngx itself, running nightly and expiring after a week (--ttl 168h), the same retention discipline you’d apply to a Restic policy outside the cluster. Schedule a whole-cluster backup too, for the case where you lose the control plane itself rather than just one namespace:
| |
Restoring
Restoring a namespace from the latest matching backup:
| |
Velero recreates the namespace’s objects and, for any volume backed up via the node-agent path, restores the volume data into a fresh PersistentVolumeClaim as part of the same operation — the pod comes up already pointing at its restored data, with no manual PVC surgery required. Watch progress the same way as a backup:
| |
Namespace migration and cluster-to-cluster DR
The same mechanism that restores a namespace into the cluster it came from will restore it into a different cluster, which is Velero’s other major use case: migrating a namespace to new hardware, or standing up a genuine disaster-recovery cluster. Point a second cluster’s Velero installation at the same backup storage location (the same S3 bucket) and any backup taken on the first cluster becomes restorable on the second:
| |
This is how I moved a workload namespace from a single-node k3s test cluster onto the multi-node cluster once it graduated from experiment to something worth running properly — backup on the old cluster, restore on the new one, verify, then decommission the old node. It is also, more importantly, the actual disaster-recovery story for “the control-plane node’s disk died”: rebuild a cluster from scratch, point Velero at the existing backup bucket, and restore everything back onto it rather than reconstructing every Deployment and Secret from memory.
What Velero doesn’t restore for you
It is worth being honest about the edges of what a Velero restore actually recreates, because assuming it covers everything is how a rehearsal fails at the worst possible moment. Velero restores API objects and, via the node-agent, volume contents — it does not restore anything that lived outside etcd and outside a PersistentVolume. Cluster-scoped resources you provisioned by hand outside of GitOps (a StorageClass, a custom admission webhook configuration, cert-manager’s ClusterIssuer objects) need to already exist on the target cluster, or need to be included explicitly in the backup and considered when restoring cluster-scoped state. Anything injected by an external system at pod-startup time — a Vault sidecar populating a Secret at runtime, a service mesh’s mutating webhook adding a proxy container — reconstructs correctly only if that external system is itself running and reachable when the restored pod starts, which on a from-scratch disaster-recovery cluster means bootstrapping those dependencies in the right order before the application restore even begins. Treat the first full restore rehearsal as the exercise that surfaces this ordering, since assuming a bare velero restore create handles it is exactly the shortcut that gets found out mid-incident.
The same caution applies to custom resources from operators. If a namespace runs a database via an operator (a PostgreSQL or Redis operator, for instance), Velero backs up the operator’s custom resources as plain Kubernetes objects, but a clean restore also needs the operator itself installed and running on the target cluster before those custom resources come back — restore the CR without the operator present, and it just sits there unreconciled, looking superficially fine while doing nothing. This is a version of the same lesson as the bare-metal restore rehearsal applied to Kubernetes: a real disaster usually trips over the dependency that quietly needed to exist first and nobody wrote down, well before it trips over the headline restore command itself.
Sizing the node-agent backup window
The file-system backup path via the node-agent DaemonSet reads volume data directly off the node’s disk while a pod is potentially still writing to it, which means backup duration scales with change rate as much as with raw data volume. A 200 GB media library that barely changes backs up quickly after the first full pass, because subsequent runs are mostly deduplicated deltas. A 50 GB database volume under constant write load can take meaningfully longer per gigabyte, both because the data is genuinely changing between chunks and because contending for the same disk I/O as the live workload slows both down. On a homelab cluster running on a handful of nodes with modest disks, this is worth planning around explicitly: schedule the heaviest-churn namespaces (anything running a database without a proper hook-based quiesce) for a window when nothing else is contending for the same disk, and keep an eye on velero backup describe for backup duration creeping upward over months as a namespace’s data grows — a job that took four minutes at initial setup and now takes forty is a useful early warning that either retention needs tightening or the node’s disk itself has become the cluster’s shared bottleneck.
Resource filtering and label selectors
Backups do not need to be all-or-nothing per namespace. Velero supports label selectors and resource-type filters for more surgical operations:
| |
Excluding events is worth doing on every backup — Kubernetes Events are high-churn, low-value for a restore, and needlessly bloat backup size and duration. A --selector restricting to labelled resources is useful for splitting a large namespace into a fast “critical tier” schedule and a slower, less frequent “everything else” schedule.
Backup hooks for databases
A filesystem-consistent snapshot of a database’s data directory is not automatically an application-consistent one — a database mid-write when the snapshot fires can restore into a state the database engine itself considers corrupt. Velero supports pre- and post-backup hooks, run as commands inside a target container, specifically for this: quiesce the database before the volume is captured, then release it afterwards.
| |
For anything running a real database inside the cluster, treat this as essential setup — pg_dump into the volume before the snapshot, or a CHECKPOINT to flush WAL to disk, is the difference between a restore that starts cleanly and one that needs the database engine’s own crash-recovery path to even come up. Test the restored database’s actual query integrity, beyond confirming the Pod reaches Running.
Choosing between the restic and kopia uploaders
The node-agent’s file-system backup path supports two upload engines: the original restic-based one, and a newer kopia-based uploader (--uploader-type=kopia at install time, or per-schedule via velero.io/data-mover-volume-mode and related annotations in newer releases). Kopia’s content-addressed chunking, the same one covered in Kopia’s dedicated writeup, generally deduplicates better across many similar PersistentVolumes — several pods running the same base image with a similar directory layout share chunks in a way restic’s older chunking is somewhat less efficient at. For a homelab with more than a handful of stateful workloads sharing a bucket, the kopia uploader is worth setting as the default; for a single namespace or two, either works fine and restic remains the better-documented option if you hit an edge case and go looking for prior art.
| |
Troubleshooting
Backups complete but restored pods stay Pending with unbound PVCs. Check that the StorageClass referenced by the original PVC still exists on the target cluster and that its provisioner is actually running — a restore does not create a StorageClass for you, only the PVC/PV objects, so a cluster migration needs the destination cluster’s storage layer set up first, matching StorageClass names.
velero backup describe shows PartiallyFailed with file-system backup errors. This usually means the node-agent DaemonSet pod on the node hosting that volume couldn’t reach the volume’s mount path, often because the pod was rescheduled mid-backup. Check velero backup logs <name> for the specific volume and pod; a retry after confirming the target pod is stable usually clears it.
Restore recreates objects but Secrets are empty or wrong. Velero backs up Secrets as regular Kubernetes objects, so if the Secret was populated by an external tool at runtime (an operator, a Vault sidecar) rather than stored directly in the cluster, the restored Secret only contains whatever was in etcd at backup time — verify secrets sourced from external systems reconcile correctly after a restore rather than assuming Velero re-triggers that provisioning.
Restore hangs indefinitely on a large volume. File-system backup and restore both run at disk and network speed through the node-agent DaemonSet; a multi-hundred-gigabyte volume over a slow link takes proportionally long. Check kubectl logs -n velero -l name=node-agent on the relevant node for progress rather than assuming a stall — genuine hangs are rarer than they look and usually resolve to “it’s still copying.”
Scheduled backups silently stop running. Confirm the velero deployment itself is healthy (kubectl get pods -n velero) — a control-plane upgrade or a node drain can leave the Velero server pod unscheduled, and a schedule with no server to execute it just accumulates missed runs with no alert unless you’re watching velero backup get or wiring alerting on top of it, the same discipline worth applying via a stack like Grafana and Prometheus.
Verdict: is it worth running?
If your cluster hosts anything you would be annoyed to rebuild by hand — and past a handful of Deployments, everything qualifies — yes. Velero is the piece that most homelab Kubernetes setups skip, because volume-level backup feels like it covers the important part, right up until a namespace gets deleted or a control-plane disk dies and you discover how much of “the application” actually lived in etcd rather than on disk. The setup cost is real but one-time: a bucket, the node-agent DaemonSet, and a couple of schedules. From there it is the same story as every other backup tool on this site — worthless until tested, so restore a namespace onto a scratch cluster at least once before you need to trust it for real.
Related Content
