GitOps with Flux: Letting Git Be Your Cluster's Source of Truth

The worst hour I ever spent on my home cluster was the one where I tried to reconstruct what was actually running on it. I’d applied a manifest months earlier, hot-patched something live during an outage, edited a config map by hand, and then forgotten all three. The YAML in my repo and the reality in the cluster had quietly diverged, and I couldn’t say with confidence what was deployed — which meant I couldn’t rebuild it if the node died. That gap between “what’s in Git” and “what’s running” is configuration drift, and it’s the single failure mode GitOps exists to kill.

The idea is simple to the point of seeming obvious once you’ve heard it: a Git repository is the single source of truth for what should be running, and an agent inside the cluster continuously reconciles reality to match it. You stop pushing changes to the cluster. Instead you push to Git, and the cluster pulls. Flux is the tool I reach for to make that happen, and after a couple of years running it at home I’d not go back.

Push versus pull, and why pull wins

It’s worth being precise about why the pull model is better, because “GitOps is good” is the kind of thing people repeat without being able to defend.

The old way is push: your laptop or a CI pipeline runs kubectl apply against the cluster. That arrangement has two problems baked in. First, something outside the cluster needs credentials with write access to production — they live in your CI system’s secret store, and they’re a juicy target. Second, nothing watches for drift between pipeline runs. If a resource is edited by hand at 2am during an incident, the push model has no idea until the next deploy, by which point everyone’s forgotten.

Flux inverts it. A controller runs inside the cluster, watches a Git repository, and applies whatever it finds there — every few minutes, forever. Three consequences fall out of this. First, no external system needs cluster credentials; the cluster reaches out, not in, which is a meaningful reduction in attack surface. Second, drift self-heals: if someone hand-edits a live resource, Flux notices it no longer matches Git and reverts it on the next reconcile. Third, your Git history is your deployment history. Want to roll back? git revert and wait sixty seconds. The same pull-and-reconcile pattern is what Argo CD offers with a dashboard you might actually use; Flux is the leaner, more CLI-and-controller-shaped take on the same idea.

Getting it bootstrapped

Flux installs itself into a cluster and, neatly, commits its own manifests to your repo so that Flux is itself managed by GitOps. The CLI does the heavy lifting:

1
2
3
4
5
6

flux bootstrap github \
  --owner=my-username \
  --repository=homelab-cluster \
  --branch=main \
  --path=./clusters/home \
  --personal

That creates a deploy key, installs the controllers into a flux-system namespace, and writes Flux’s own config under clusters/home/. From this moment, the repo runs the cluster — including Flux itself, which means upgrading Flux is also just a commit. One practical note: the command needs a GitHub token (via the GITHUB_TOKEN environment variable) with permission to create the repo and a deploy key, so generate a fine-grained personal access token scoped to that one repository rather than reaching for a classic all-powerful token.

Telling Flux what to watch

Flux’s model is two kinds of object. A source says where to find config — a Git repository, an OCI artifact, a Helm repository, or an S3-compatible bucket. A Kustomization (or HelmRelease) says what to do with it. Here’s a source pointing at an app repo, paired with a Kustomization that applies it:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
  name: my-app
  namespace: flux-system
spec:
  interval: 1m
  url: https://github.com/my-username/my-app
  ref:
    branch: main
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: my-app
  namespace: flux-system
spec:
  interval: 5m
  sourceRef:
    kind: GitRepository
    name: my-app
  path: "./deploy"
  prune: true

The two interval fields do different jobs: the source’s interval is how often Flux re-fetches the repo, and the Kustomization’s is how often it re-applies and re-checks for drift even if the source hasn’t changed. That second one is what makes drift self-heal on a timer rather than only on a git push.

That prune: true is the quiet hero. When you delete a manifest from Git, Flux deletes the corresponding resource from the cluster. No orphaned objects lingering because someone forgot to kubectl delete them. The repo is genuinely the truth, additions and removals alike. Leave it off and Flux only ever adds — a perfectly valid choice while you’re learning, but it undercuts half the point.

You can watch reconciliation happen, or force it when you’re impatient:

1
2

flux get kustomizations
flux reconcile kustomization my-app --with-source

Helm without the imperative bits

Plenty of the software you’ll run ships as a Helm chart, and the instinct is to helm install it — which puts you straight back in the push model, with chart state living in the cluster instead of in Git. Flux’s answer is the HelmRelease: you declare a Helm repository as a source and a release as a desired-state object, and the helm-controller reconciles it the same way the kustomize-controller handles plain manifests.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

apiVersion: source.toolkit.fluxcd.io/v1
kind: HelmRepository
metadata:
  name: ingress-nginx
  namespace: flux-system
spec:
  interval: 1h
  url: https://kubernetes.github.io/ingress-nginx
---
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: ingress-nginx
  namespace: flux-system
spec:
  interval: 30m
  chart:
    spec:
      chart: ingress-nginx
      version: "4.x"
      sourceRef:
        kind: HelmRepository
        name: ingress-nginx
  values:
    controller:
      replicaCount: 2

The values block is your values.yaml, in Git, version-controlled with everything else. Pin the chart version — a floating version means a chart-repo update can silently change what’s running, which is exactly the drift you adopted GitOps to escape. This is also the clean way to manage things you’d otherwise install by hand and forget, which pairs neatly with understanding what Helm charts actually do and when to skip them.

Structuring the repo so it scales

A single flat directory of manifests works for an afternoon and then starts to hurt. The layout I’ve settled on, and the one Flux’s own examples nudge you towards, separates infrastructure (ingress, cert-manager, storage classes — the things apps depend on) from apps, and uses a per-cluster directory at the top so one repo can describe more than one cluster:

1
2
3
4
5
6
7
8
9

clusters/
  home/                 # entrypoint: Kustomizations pointing at the dirs below
    infrastructure.yaml
    apps.yaml
infrastructure/
  ingress-nginx/
  cert-manager/
apps/
  my-app/

The trick that keeps this sane is dependsOn: the apps Kustomization declares dependsOn: [infrastructure], so Flux won’t try to deploy an app that needs an ingress controller before the ingress controller exists. Ordering becomes declarative instead of a sequence of commands you have to remember to run in the right order — which is the whole spirit of the thing.

The honest friction

GitOps is not free of sharp edges, and pretending otherwise just sets people up to be annoyed.

The biggest is secrets. You absolutely cannot commit plaintext secrets to Git, so you need SOPS with an age or KMS key, or the Sealed Secrets controller, to encrypt them before they land in the repo. Setting that up is the first real hurdle, and it’s non-negotiable. My pick for a homelab is SOPS with age: a single keypair, the private half stored as a Kubernetes secret that Flux decrypts with, and encrypted values that are safe to sit in a public repo. The wiring is a .sops.yaml in the repo that tells SOPS to encrypt only the data and stringData fields of secrets (so the resource is still readable in a diff, just with the values scrambled), plus a one-line decryption block on the Kustomization pointing at the in-cluster key:

1
2
3
4
5

spec:
  decryption:
    provider: sops
    secretRef:
      name: sops-age

The payoff is that a leaked repo leaks nothing usable, and rotating a secret is a commit like any other. Budget an evening for it and don’t deploy anything real until it works.

Troubleshooting: where deploys actually break

The second friction is a debugging shift, and it trips up everyone coming from kubectl apply. When a deploy fails, the error isn’t on your terminal — it’s buried in a controller’s status. The reflex to learn:

1
2
3
4
5
6
7
8

# Is the source even fetching? Watch the READY/STATUS columns.
flux get sources git
# Is the Kustomization applying? This is where most failures surface.
flux get kustomizations
# The full story, including the YAML build/apply error:
kubectl -n flux-system describe kustomization my-app
# Tail the controller itself when status isn't enough:
kubectl -n flux-system logs deploy/kustomize-controller

The failures I hit most often: a Kustomization stuck False with a build error usually means a bad path or a kustomization.yaml that references a file that isn’t there. A source stuck not-ready is almost always auth — a revoked deploy key or a token that expired. And a resource that “won’t update” despite a fresh commit is often a dependsOn ordering issue, where one Kustomization is waiting on another that’s failing silently upstream. flux get kustomizations showing the whole dependency chain at once is how you spot it.

The third friction is discipline, and it’s a people problem, not a tooling one. GitOps only works if everyone stops touching the cluster directly. The first time someone fixes an outage with a live kubectl edit, Flux will cheerfully revert their fix on the next reconcile — which is either exactly what you wanted or a baffling regression, depending on whether the team has actually bought in. The fix isn’t technical; it’s agreeing that the repo is the truth and meaning it.

The verdict

For a serious cluster — anything you’d be upset to lose and have to rebuild from memory — GitOps with Flux is worth every hour of setup. The payoff is enormous: a cluster you can recreate from an empty machine and a Git URL, a deployment history you can audit, and drift that fixes itself. I run it at home precisely because I forget what I did six months ago, and Flux remembers for me. It also composes well with the rest of a cluster — once you’ve added a second machine to your cluster, having Git as the single definition of what runs where stops the “which node was that on again?” question dead.

For a throwaway test cluster you’ll delete on Friday, it’s overkill; just kubectl apply and move on. The setup cost is real and the secrets dance is fiddly, and none of that pays off on a cluster you won’t miss. But the moment a cluster crosses from “experiment” to “thing I depend on,” let Git be the truth. The break-even point arrives faster than you’d expect — usually the first time you can’t remember what you changed last week. Your future self, staring at a dead node and wondering what was supposed to be running, will thank you for the boring repo that answers the question in full.