← All posts

The first secret is the one you can't commit

A single glowing seed-shaped key at the base of a dark circuit-board tree whose traces branch upward into hundreds of faint nodes, all drawing their glow from that one source.

Every “we do GitOps properly” story has a chicken-and-egg problem hiding in the first paragraph, and most posts quietly skip it.

The pitch is clean: secrets never live in git. Instead, the cluster runs the External Secrets Operator (ESO), which reads from a real secret store — here, a self-hosted Infisical — and materialises a Kubernetes Secret for each app. Your repo only ever contains an ExternalSecret manifest: a pointer (“give immich the value at key IMMICH_DB_PASSWORD”), never a value. Beautiful. Auditable. Diff-able.

Except — ESO has to authenticate to Infisical somehow. That’s a credential. Where does it live?

Secret zero

You can’t store the bootstrap credential in git (that’s the whole point), and you can’t fetch it from the secret store (you need it to reach the secret store). This is secret zero: the one credential that the entire chain hangs off, that has to be injected from outside the GitOps loop.

So you treat it as exactly that — special, minimal, and out-of-band:

  • It’s a machine identity scoped to read-only, and scoped per cluster. cave’s ESO can’t read alfred’s secrets and vice-versa. A leak is blast-radius-limited to one cluster’s read path.
  • It’s applied once, by hand (or by a sealed bootstrap step), never committed. Everything downstream of it is declarative.
  • After it lands, a single ClusterSecretStore object — which is in git, because it’s just a pointer to the store plus a reference to secret zero — wires the whole cluster up.

From there the dam breaks in the good way: every app’s ExternalSecret resolves through that store, ESO keeps the Secret in sync, and your repo stays a map of names, not values.

Secret-zero bootstrap chain

The discipline around the one exception

One out-of-band credential is fine. The trap is letting it sprawl — or fumbling its rotation. Two rules earn their keep:

Per-cluster, least-privilege identities. It’s tempting to mint one powerful identity and reuse it everywhere. Don’t. A read-only identity per cluster means the bootstrap secret can read and nothing else, in one place. The write path (provisioning new secrets) is a separate, rarely-used identity that doesn’t sit on every node.

Never revoke a shared identity until every consumer has moved off it. This sounds obvious and is the single easiest way to take the fleet down. When you split a shared identity into per-cluster ones, the old one stays valid until you’ve verified each cluster is happily authenticating on its own — then, and only then, you revoke. Revoke-first-verify-later turns a tidy-up into an outage.

What still belongs in git (encrypted)

A purist would stop here, but reality has a few things that genuinely need to live in the repo — a value a bootstrap step reads before ESO is even running. For those, the answer isn’t “commit it in plaintext and feel bad,” it’s SOPS + age: the value is encrypted in git, decryptable only by a key that lives on the operators’ machines (and the cluster), never in the repo. Same principle as secret zero — the decryption key is the out-of-band thing — applied to the handful of values that can’t wait for the operator to spin up.

The shape of it

The win isn’t any one tool. It’s the shape: exactly one credential lives outside GitOps, it’s read-only and per-cluster, and it’s the seed the whole tree grows from. Everything else is a pointer you can show your worst enemy. When someone asks “where are your secrets?”, the honest answer is “in the store — the repo just knows their names,” and that one sentence is the whole security model.

This runs across a five-cluster Talos homelab; the ESO + Infisical wiring, the per-cluster read-only identities, and the revoke-order discipline are the load-bearing pieces.