Persistent Volumes in K3s Without Losing Your Data

The first time a Kubernetes pod ate my data I was running a small Postgres instance on a single-node K3s box, doing exactly the sort of casual maintenance that gets you into trouble. I deleted a Helm release to reinstall it cleanly, watched the pod terminate, and only afterwards noticed that the PersistentVolumeClaim had gone with it, and the volume behind it, and every row inside. The database was five minutes of test data, so the loss cost me nothing except the useful fright. It taught me the single most important thing about storage on Kubernetes: the platform will keep your data exactly as safely as you have explicitly told it to, and the defaults are tuned for throwaway workloads that would happily be recreated from scratch. The family photo library you just migrated off a NAS wants something sturdier.

That gap between what the defaults do and what a homelabber assumes they do is where almost every “K3s ate my data” story comes from. So before any YAML, the why.

Why a pod cannot be trusted with your bytes

A pod is designed to be disposable. That is the entire premise of the scheduler: a pod is a unit of work that can be killed, moved to another node, restarted, scaled to zero and back, or replaced wholesale during a rolling update, and the system should keep humming through all of it. Anything written to a container’s own filesystem lives and dies with that container — recreate the pod and you get a pristine image again, with none of the writes the old one made. For a stateless web frontend that behaviour is precisely what you want. For anything that remembers things between restarts it is a disaster waiting for a convenient moment.

The mechanism that decouples the data lifecycle from the pod lifecycle is the PersistentVolumeClaim. A PVC is a request that says “I need 10Gi of storage that survives me.” Kubernetes satisfies that request by binding it to a PersistentVolume, the actual chunk of real storage. When the pod is rescheduled onto another node, or deleted and recreated by a Deployment, the new pod mounts the same PVC and finds its files exactly where the old one left them. The data outlives any individual pod because it was never the pod’s to begin with. If you have used Docker Compose, this is the same instinct behind a named volume rather than a bind mount into the container — I wrote about the durable version of that habit in Docker Compose patterns that age well — but Kubernetes splits the idea into more moving parts, and each part has a default you should know about.

Three objects form the triangle. A StorageClass describes how storage gets provisioned — which provisioner to call, what parameters to hand it, what reclaim policy the resulting volume inherits. A PVC is the request a workload makes: this much space, this access mode, optionally from this named StorageClass. A PersistentVolume is the thing itself, the provisioned disk or directory. With dynamic provisioning, which is the normal case on K3s, you never hand-write a PV: you create a PVC, the StorageClass’s provisioner creates a matching PV on demand and binds the two together, and your pod mounts the claim. It feels like magic right up until the reclaim policy fires.

What K3s gives you out of the box, and its sharp edges

K3s ships with storage that works on the first try, which is a large part of why it is such a pleasant thing to run at home. The default StorageClass is local-path, backed by Rancher’s local-path-provisioner. When a PVC lands, the provisioner picks the node where the pod is scheduled and carves out a directory under /var/lib/rancher/k3s/storage/ on that node’s local disk. No configuration, no external dependency, no NFS box humming in the cupboard. For a single-node cluster it is genuinely all you need.

The sharp edges appear the moment you have more than one node. local-path is node-local by definition: the data physically sits on one machine’s disk. It is ReadWriteOnce, so exactly one node may mount it at a time. It has no replication, so if that node’s disk dies the data dies with it. And a local-path volume cannot be expanded — the provisioner does not support online resize, so a claim that turns out too small becomes a migration job rather than a one-line patch. All of this is by design: a deliberately minimal provisioner doing a minimal, reliable job. The trap is assuming it behaves like the fancy cloud storage classes you may have met on a managed cluster, where volumes float freely between nodes and grow on demand.

Then there is the reclaim policy, which is where my Postgres story lives. Dynamically provisioned volumes inherit their StorageClass’s reclaimPolicy, and for local-path — as for most dynamic provisioners — that default is Delete. Reclaim policy governs what happens to the PV when its PVC is deleted. Under Delete, removing the PVC tells the provisioner to remove the underlying volume too, directory and all, immediately and without ceremony. Under Retain, deleting the PVC leaves the PV in place, marked Released, holding your data untouched until you deal with it by hand. The default optimises for tidy clusters that never leak orphaned disks. It does not optimise for the human who runs helm uninstall at half eleven at night and expects the database to still be there tomorrow.

The correct habit is to decide, per workload, whether the data is precious, and to put anything precious on a StorageClass with reclaimPolicy: Retain. You cannot edit the reclaim policy of the built-in local-path class cleanly (K3s manages it and will revert edits on restart), so the tidy approach is a second StorageClass of your own that points at the same provisioner with the policy you actually want.

The YAML: a Retain StorageClass, a claim, and a StatefulSet

Here is the shape I reach for. A custom StorageClass that reuses the local-path provisioner but flips the reclaim policy to Retain, a PVC that asks for storage from it, and a StatefulSet that provisions a stable per-replica volume through volumeClaimTemplates — the right pattern for anything database-shaped, because each replica gets its own durable PVC with a predictable name that follows it across reschedules.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67

# A StorageClass that keeps the volume when the PVC is deleted.
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: local-path-retain
provisioner: rancher.io/local-path
reclaimPolicy: Retain          # deleting the PVC leaves the PV + data behind
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: false    # local-path can't expand; be honest about it
---
# A standalone claim that uses it.
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: photos-data
  namespace: media
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: local-path-retain
  resources:
    requests:
      storage: 20Gi
---
# A StatefulSet giving each replica its own stable, durable PVC.
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: postgres
  namespace: media
spec:
  serviceName: postgres
  replicas: 1
  selector:
    matchLabels:
      app: postgres
  template:
    metadata:
      labels:
        app: postgres
    spec:
      securityContext:
        fsGroup: 999            # postgres group in the official image
      containers:
        - name: postgres
          image: postgres:17
          env:
            - name: POSTGRES_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: postgres-secret
                  key: password
            - name: PGDATA
              value: /var/lib/postgresql/data/pgdata
          volumeMounts:
            - name: pgdata
              mountPath: /var/lib/postgresql/data
  volumeClaimTemplates:
    - metadata:
        name: pgdata
      spec:
        accessModes:
          - ReadWriteOnce
        storageClassName: local-path-retain
        resources:
          requests:
            storage: 20Gi

The volumeClaimTemplates block is the part worth internalising. A StatefulSet stamps out one PVC per replica, named deterministically — pgdata-postgres-0 here — and re-attaches that same claim whenever the pod restarts or reschedules. Scale to three replicas and you get pgdata-postgres-0, -1, -2, each an independent volume. That stable identity is exactly what stateful software wants, and it is why you use a StatefulSet for databases rather than bolting a single shared PVC onto a Deployment.

One practical note on where these manifests should live: in Git, applied by a controller, never kubectl apply-ed from your laptop and forgotten. The PVC definitions above are ordinary declarative objects and belong in the same reconciled repository as everything else, which is the whole argument I made in A homelab GitOps loop with Flux and Gitea — a StorageClass you created by hand a year ago and can’t reproduce is its own small disaster.

When one node stops being enough

local-path runs out of road the day a pod needs to move to a different node and take its data along, or the day two pods on different nodes need the same volume at once. Access modes describe the second problem precisely. ReadWriteOnce (RWO) means one node may mount the volume read-write. ReadOnlyMany (ROX) means many nodes may mount it read-only. ReadWriteMany (RWX) means many nodes may mount it read-write simultaneously — and local-path simply cannot offer RWX, because a directory on one node’s disk has no way to appear on another node. The moment your architecture needs shared read-write storage across nodes, you have outgrown the default provisioner.

There are three honest upgrade paths for a homelab. Longhorn is the one I run now: replicated distributed block storage from the Rancher stable, which keeps multiple copies of each volume across nodes so a dead disk costs you nothing, offers RWX through an internal NFS export, takes snapshots, runs scheduled backups off to S3 or an NFS target, and gives you a genuinely usable web UI to watch it all. It asks for more RAM and more disk overhead than local-path, and it wants healthy networking between nodes, but for replicated storage that survives a node failure it is the natural next step. The NFS subdirectory external provisioner is the simpler option: point it at an existing NFS server and it hands out RWX volumes as subdirectories, which is wonderfully uncomplicated as long as you accept that the single NFS box is one power supply away from taking every volume offline at once. hostPath pinned to a node is the crude fallback — nail a pod to one machine with a node selector and mount a real path on it — and it works until the day that node is down and the pod cannot be scheduled anywhere else, at which point the crudeness bites.

Whichever you land on, the backup rule is the same and it is the one people skip: a snapshot living on the same disk as the volume is not a backup. It rescues you from a bad DROP TABLE and does absolutely nothing the day the drive itself dies. Longhorn’s recurring jobs can snapshot and ship a real backup to off-box storage on a schedule; if you would rather stay tool-agnostic, restic or Velero can back the volumes up to somewhere the cluster cannot set on fire. Configure the off-box copy before you trust the cluster with anything you would grieve.

Troubleshooting the usual failures

A PVC stuck Pending forever. This is the most common one and the diagnostic is always the same first move: kubectl get pv,pvc -A to see the state, then kubectl describe pvc <name> to read the events at the bottom, which almost always name the cause outright. The usual culprits are no default StorageClass at all (nothing to satisfy a claim that didn’t name one), a storageClassName with a typo that matches no class, or the provisioner pod not running. With volumeBindingMode: WaitForFirstConsumer — which local-path uses — a PVC also stays deliberately Pending until a pod actually consumes it, so a claim with no workload attached yet is behaving exactly as designed and will bind the instant a pod turns up to use it.

A pod stuck ContainerCreating with a multi-attach error. You will see Multi-Attach error for volume ... Volume is already exclusively attached to one node. An RWO volume can only be mounted on one node, and after a pod reschedules onto a new node the old attachment sometimes hasn’t been released yet, so the new pod waits. It usually clears itself once the old node’s mount detaches. If the pod moved because a node went down hard, the attachment can hang until you intervene. The nastier variant is specific to local-path: the pod rescheduled onto a node where the data does not physically exist, because the directory only ever lived on the original node’s disk. No amount of waiting fixes that one — the fix is to pin the workload back to the node that holds the data, or to have been on replicated storage in the first place.

Recovering data after a Delete scare, and patching to Retain. For an existing volume you want to protect, patch the live PV: kubectl patch pv <pv-name> -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}'. Now deleting the PVC leaves the PV as Released rather than deleting it. To re-bind a Released PV to a fresh PVC, you have to clear the stale claim reference first — kubectl patch pv <pv-name> -p '{"spec":{"claimRef":null}}' — after which the PV returns to Available and a new PVC of matching size and class can bind to it and mount the old data. Practise this on a calm afternoon so you already know the dance by the time an outage forces it on you.

Permission denied on the mounted volume. A container running as a non-root user often can’t write to a freshly provisioned volume owned by root. The clean fix is securityContext.fsGroup on the pod, as in the StatefulSet above: Kubernetes chowns the volume to that group ID at mount time so the process can write. It saves you the ugly workaround of an init container running chown by hand.

A volume that turned out too small. On a StorageClass that supports it you would edit the PVC’s requested size and let it expand online. On local-path you cannot — there is no online resize — so the honest path is migration: provision a larger PVC, copy the data across (a throwaway pod mounting both claims and running cp -a does the job), then repoint the workload. Annoying, entirely avoidable by sizing generously up front.

The verdict

K3s storage is excellent for what it is, provided you meet it on its own terms. On a single node, local-path with a Retain StorageClass for the things you care about is a genuinely solid setup that I would recommend without hesitation — cheap, fast, dependency-free, and safe once you have flipped the one reclaim-policy switch that the defaults get wrong for stateful work. The failure mode is treating the defaults as if someone had already made the durability decisions for you; they haven’t, and Delete is waiting.

Add a second node and the calculus changes. The moment you want workloads to migrate freely or share storage across machines, budget the afternoon it takes to stand up Longhorn, get its backups shipping off-box, and confirm you can actually restore from them. The restore rehearsal is the part that matters — an untested backup is a rumour. Do that once, deliberately, while nothing is on fire, and K3s becomes a storage platform you can trust with the photo library after all.