Kubernetes Secrets Management: SOPS, Sealed Secrets, or External Secrets

Kubernetes Secrets are not secret. That’s the first thing nobody tells you. A Secret object is base64-encoded YAML sitting in etcd, and base64 is encoding, not encryption — anyone with get secrets on the namespace can read it back in plaintext with a single command. Encryption at rest in etcd helps a little, but the real problem turns up the moment you adopt GitOps: now you want everything in a repo, and committing a base64 blob of your database password to Git is the kind of decision that ends careers. So you reach for tooling. There are three serious contenders, and I’ve run all of them in anger.

1 SOPS: encrypt the file, commit the file

SOPS (Secrets OPerationS, originally from Mozilla) encrypts the values in a YAML/JSON file while leaving the keys readable, so a diff still tells you which secret changed without leaking what it changed to. It backs onto a KMS — age, GPG, AWS KMS, GCP KMS, Vault. With age it’s almost embarrassingly simple.

# Generate a key, encrypt a manifest in place
age-keygen -o age.key
export SOPS_AGE_RECIPIENTS=$(grep public age.key | cut -d' ' -f4)

sops --encrypt --encrypted-regex '^(data|stringData)$' \
  secret.yaml > secret.enc.yaml

The resulting secret.enc.yaml is safe to commit. In the cluster you decrypt at apply time. Flux has native SOPS support; for Argo or plain kubectl you use the ksops plugin or helm-secrets. The win is that the source of truth is the encrypted file in Git and the decryption key lives only in the cluster (as a Secret, yes — chicken and egg, but a single bootstrap secret instead of fifty).

2 Sealed Secrets: a controller does the unsealing

Bitnami’s Sealed Secrets flips the model. You run a controller in-cluster that holds a private key. You encrypt against its public key with the kubeseal CLI, producing a SealedSecret custom resource. Only that specific controller, in that specific namespace, can decrypt it — encryption is scoped to the name and namespace by default, so you can’t lift a sealed secret into another namespace.

kubectl create secret generic db-creds \
  --from-literal=password='hunter2' \
  --dry-run=client -o yaml \
| kubeseal --controller-namespace kube-system \
  --format yaml > db-creds-sealed.yaml

kubectl apply -f db-creds-sealed.yaml

The controller watches SealedSecret objects and materialises a real Secret next to each one. The catch is the private key: it’s generated in-cluster and rotated automatically, which means you must back it up. Lose that key and every sealed secret in Git becomes undecryptable ciphertext — there is no recovery, no support line, just a repo full of useless blobs and a cluster you now have to re-seed from scratch. I learned this the way you’d expect, on a cluster I’d casually torn down to “rebuild cleanly”. Export the sealing key to your password manager the moment you install the controller, and treat it like a root CA.

3 External Secrets: don’t store secrets at all

The External Secrets Operator (ESO) takes the position that secrets shouldn’t live in your cluster or your repo. They live in a real secrets manager — Vault, AWS Secrets Manager, GCP Secret Manager, Azure Key Vault, 1Password — and ESO syncs them in. Git holds only a reference.

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: db-creds
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: vault-backend
    kind: ClusterSecretStore
  target:
    name: db-creds          # the k8s Secret it creates
  data:
    - secretKey: password
      remoteRef:
        key: secret/data/prod/db
        property: password

Nothing sensitive is in that manifest — it’s a pointer. Rotation happens in your secrets manager and propagates on the refresh interval. This is the right answer at scale, and the wrong answer if you don’t already operate a secrets manager, because now you’re running and securing Vault as well.

4 Which one, then

I pick by where the source of truth wants to live:

• Small homelab or a team that already lives in Git — SOPS with age. No extra controller, plaintext-free diffs, trivial to reason about. My default.
• You want Git to be the source of truth but hate KMS plumbing — Sealed Secrets. One controller, kubeseal, done. Just automate the key backup before you ship anything.
• You already run Vault/cloud secret managers, or have compliance breathing down your neck — External Secrets. Rotation and audit come for free; the cost is operating the backend.

Is any of this worth it over raw kubectl create secret? If you’re doing GitOps, absolutely — the alternative is secrets that exist only in someone’s terminal history and a cluster nobody can rebuild. If you’re a single operator clicking around kubectl by hand, SOPS gives you 90% of the benefit for an afternoon’s setup. Don’t reach for External Secrets until you genuinely have a secrets manager to point it at; running Vault “to be tidy” is how a weekend project becomes a second job.