Kubernetes Secrets Management: SOPS, Sealed Secrets, or External Secrets

Kubernetes Secrets are not secret. That’s the first thing nobody tells you. A Secret object is base64-encoded YAML sitting in etcd, and base64 is encoding, not encryption — anyone with get secrets on the namespace can read it back in plaintext with a single command. If you doubt it, kubectl get secret db-creds -o jsonpath='{.data.password}' | base64 -d will happily print your database password to the terminal of anyone with the right RBAC. Encryption at rest in etcd helps a little, but the real problem turns up the moment you adopt GitOps: now you want everything in a repo, and committing a base64 blob of your database password to Git is the kind of decision that ends careers. It doesn’t even need a breach — a public mirror, a fork, a laptop backup that syncs to someone’s personal cloud, and the credential is loose forever, because Git never forgets. So you reach for tooling. There are three serious contenders, and I’ve run all of them in anger on my own cluster.

Before the tour, one principle that cuts through the choice: decide where the source of truth wants to live. In Git as ciphertext? In the cluster behind a controller? In a dedicated secrets manager? Each tool answers that question differently, and picking the wrong answer for your setup is how you end up fighting your own tooling for the next year.

SOPS: encrypt the file, commit the file

SOPS (Secrets OPerationS, originally from Mozilla) encrypts the values in a YAML/JSON file while leaving the keys readable, so a diff still tells you which secret changed without leaking what it changed to. It backs onto a KMS — age, GPG, AWS KMS, GCP KMS, Vault. With age it’s almost embarrassingly simple.

1
2
3
4
5
6

# Generate a key, encrypt a manifest in place
age-keygen -o age.key
export SOPS_AGE_RECIPIENTS=$(grep public age.key | cut -d' ' -f4)

sops --encrypt --encrypted-regex '^(data|stringData)$' \
  secret.yaml > secret.enc.yaml

The resulting secret.enc.yaml is safe to commit. In the cluster you decrypt at apply time. Flux has native SOPS support; for Argo or plain kubectl you use the ksops plugin or helm-secrets. The win is that the source of truth is the encrypted file in Git and the decryption key lives only in the cluster (as a Secret, yes — chicken and egg, but a single bootstrap secret instead of fifty).

What the encrypted file actually looks like matters, because it’s the ergonomic selling point. Only the values are ciphertext; the keys, and a sops metadata block recording which recipients can decrypt it, stay readable:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

apiVersion: v1
kind: Secret
metadata:
  name: db-creds
stringData:
  password: ENC[AES256_GCM,data:9Kf2...,tag:xQ==,type:str]
sops:
  age:
    - recipient: age1ql3z7...
      enc: |
        -----BEGIN AGE ENCRYPTED FILE-----
        ...
        -----END AGE ENCRYPTED FILE-----
  encrypted_regex: ^(data|stringData)$

Git diffs stay meaningful — you can see that password changed without seeing the value — and reviewers can approve a rotation without ever holding the plaintext. The catch is that decryption is manual-ish: someone or something has to hold the age key, and if that key leaks, every secret encrypted to it is exposed. Treat the age private key like an SSH host key. On the topic of which, if you’re still generating RSA keys anywhere in this stack, switch to age (or ssh-keygen -t ed25519 for SSH) and be done with it.

Sealed Secrets: a controller does the unsealing

Bitnami’s Sealed Secrets flips the model. You run a controller in-cluster that holds a private key. You encrypt against its public key with the kubeseal CLI, producing a SealedSecret custom resource. Only that specific controller, in that specific namespace, can decrypt it — encryption is scoped to the name and namespace by default, so you can’t lift a sealed secret into another namespace.

1
2
3
4
5
6
7

kubectl create secret generic db-creds \
  --from-literal=password='hunter2' \
  --dry-run=client -o yaml \
| kubeseal --controller-namespace kube-system \
  --format yaml > db-creds-sealed.yaml

kubectl apply -f db-creds-sealed.yaml

A detail that trips people up: the sealed secret is not portable. Because encryption is scoped to name and namespace, you cannot copy a SealedSecret from one namespace to another and expect it to unseal — the controller will refuse. You can loosen this to namespace-wide or cluster-wide scope with the sealedsecrets.bitnami.com/namespace-wide or .../cluster-wide annotations, but the default strict scoping is a security feature, not an inconvenience: it stops a leaked sealed secret being replayed into a namespace the attacker controls. Keep the default unless you have a concrete reason not to.

The controller watches SealedSecret objects and materialises a real Secret next to each one. The catch is the private key: it’s generated in-cluster and rotated automatically, which means you must back it up. Lose that key and every sealed secret in Git becomes undecryptable ciphertext — there is no recovery, no support line, just a repo full of useless blobs and a cluster you now have to re-seed from scratch. I learned this the way you’d expect, on a cluster I’d casually torn down to “rebuild cleanly”. Export the sealing key to your password manager the moment you install the controller, and treat it like a root CA:

1
2
3
4

# Back up the sealing key set — do this before you seal anything you can't regenerate
kubectl get secret -n kube-system \
  -l sealedsecrets.bitnami.com/sealed-secrets-key \
  -o yaml > sealed-secrets-key-backup.yaml

Store that file somewhere offline and encrypted; it is the one artefact that stands between you and a full re-seed. A note on provenance, since it comes up: despite the wider 2025 upheaval around Bitnami’s container catalogue, the Sealed Secrets project (now maintained under bitnami-labs/sealed-secrets) kept its own release cadence and image distribution, so it remains actively maintained and unaffected. It’s not abandonware; it’s just quietly boring, which is exactly what you want from something guarding your credentials.

External Secrets: don’t store secrets at all

The External Secrets Operator (ESO) takes the position that secrets shouldn’t live in your cluster or your repo. They live in a real secrets manager — Vault, AWS Secrets Manager, GCP Secret Manager, Azure Key Vault, 1Password — and ESO syncs them in. Git holds only a reference.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: db-creds
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: vault-backend
    kind: ClusterSecretStore
  target:
    name: db-creds          # the k8s Secret it creates
  data:
    - secretKey: password
      remoteRef:
        key: secret/data/prod/db
        property: password

Nothing sensitive is in that manifest — it’s a pointer. Rotation happens in your secrets manager and propagates on the refresh interval. This is the right answer at scale, and the wrong answer if you don’t already operate a secrets manager, because now you’re running and securing Vault as well.

The under-appreciated benefit of ESO is rotation without redeploys. Change the password in Vault and, within one refresh interval, the in-cluster Secret updates itself — no Git commit, no pipeline run, no kubectl apply. Pair it with a workload that reloads credentials on the fly (or a controller that restarts pods when the Secret changes) and you have genuine automated rotation, which is the thing every compliance auditor asks for and almost nobody actually has. The cost, again, is that you now own the backend: Vault needs unsealing, backing up, upgrading, and monitoring, and if it’s down your whole cluster’s secret plane is down with it. That’s not a reason to avoid ESO — it’s a reason to be honest about whether you’re signing up to run a secrets manager as a first-class service or bolting one on as an afterthought.

Troubleshooting the three of them

Each tool has a signature failure that will eat an evening if you don’t recognise it.

SOPS: “no matching creds found” on decrypt. The cluster’s age key doesn’t match the recipients the file was encrypted to. This happens after you rotate the age key but forget to re-encrypt existing files, or when a teammate encrypts to their key and not the cluster’s. Run sops --decrypt locally to confirm the file is sound, then check that the in-cluster key Secret actually contains the private half of a recipient listed in the file’s sops block.

Sealed Secrets: SealedSecret applies but no Secret appears. Nine times out of ten the sealed secret was encrypted against a different controller — you moved clusters, restored from a backup, or sealed with the wrong --controller-namespace. Sealed Secrets bind ciphertext to the controller’s key and to the target name/namespace, so a secret sealed for apps/db-creds will silently refuse to unseal into staging/db-creds. Check the controller logs: kubectl logs -n kube-system deploy/sealed-secrets-controller prints the exact reason.

External Secrets: ExternalSecret stuck SecretSyncedError. Almost always auth to the backend — an expired Vault token, a misconfigured ClusterSecretStore, or an IAM role that can’t read the path. kubectl describe externalsecret db-creds shows the underlying error verbatim. The trap is the refresh interval hiding the problem: a secret that synced once and then broke keeps serving the last good value until something restarts, so a green pod can be running on a credential the operator can no longer refresh. Watch the status conditions, not just whether the app is up.

Whichever you pick, keep it inside a broader habit of knowing what your cluster is doing. Secrets tooling fails quietly, and the failure surfaces as an app that can’t reach the resource it depends on. The bootstrap decryption key, the sealing key, the Vault token — these are exactly the stateful, must-survive artefacts that argue for putting your cluster’s important state on real storage rather than an ephemeral node disk; my take on that is in Longhorn vs OpenEBS for Kubernetes storage. And a decrypt loop that thrashes because a controller keeps getting OOM-killed will look exactly like a secrets bug, so make sure the controller has sane resource requests and limits before you go blaming the encryption.

Which one, then

I pick by where the source of truth wants to live:

• Small homelab or a team that already lives in Git — SOPS with age. No extra controller, plaintext-free diffs, trivial to reason about. My default.
• You want Git to be the source of truth but hate KMS plumbing — Sealed Secrets. One controller, kubeseal, done. Just automate the key backup before you ship anything.
• You already run Vault/cloud secret managers, or have compliance breathing down your neck — External Secrets. Rotation and audit come for free; the cost is operating the backend.

Is any of this worth it over raw kubectl create secret? If you’re doing GitOps, absolutely — the alternative is secrets that exist only in someone’s terminal history and a cluster nobody can rebuild. If you’re a single operator clicking around kubectl by hand, SOPS gives you 90% of the benefit for an afternoon’s setup. Don’t reach for External Secrets until you genuinely have a secrets manager to point it at; running Vault “to be tidy” is how a weekend project becomes a second job.