~/jwright
← All posts
·4 min read

The cert that couldn't see its own proof

A sealed structure split by a translucent wall: a glowing proof-token sits in the open on the outside where a distant beam finds it, while an identical vantage on the inside looks toward the same spot and sees nothing.

Here’s the uncomfortable default: cert-manager checks its own homework through whatever DNS the cluster hands it. On a normal network that’s fine — the cluster’s resolvers see the same internet everyone else does. On a split-horizon network, they don’t. Your internal DNS is authoritative for the domain, answers first, and knows nothing about the record you just published to the world.

So the DNS-01 challenge does something quietly absurd. cert-manager writes the _acme-challenge TXT record to Cloudflare — and it lands. dig @1.1.1.1 returns the token. Let’s Encrypt could validate it right now. But before asking the CA to look, cert-manager runs a self-check: it resolves the record itself, through cluster DNS, through the Pi-hole that owns bztmon.org internally and has no such TXT record. Nothing comes back. So it waits, and checks again, and loops on "not yet propagated" — forever.

It’s posting a letter into the public postbox, then walking back inside to check the hallway mail slot for it. The letter is out there. The postman can see it. You’re looking in the one box it will never reach — and concluding it hasn’t been sent.

The self-check resolving through the wrong horizon while the CA sees the record

Look where the examiner looks

The fix is one Helm value, and it encodes the whole lesson:

dns01RecursiveNameservers: "1.1.1.1:53,8.8.8.8:53"
dns01RecursiveNameserversOnly: true

That points the self-check at public resolvers — the same vantage Let’s Encrypt validates from — instead of the cluster’s own view of the world. The moment cert-manager checks the postbox the postman actually collects from, the challenge that had been pending for an age clears in seconds. No record changed. No DNS changed. Only the observer moved.

One footnote if you run default-deny egress: the check now leaves the cluster on port 53, so world:53 has to be allowed for the cert-manager pods. Here it already was, courtesy of the same policy that lets cloudflared out.

Two proofs, one postbox

The same challenge has a second way to hang, and it looks identical from the outside. Request a certificate for both the wildcard *.bztmon.org and the apex bztmon.org, and Let’s Encrypt opens two authorisations — both of which validate at the same record name, _acme-challenge.bztmon.org, with different tokens. cert-manager’s Cloudflare solver dutifully writes one, then the other solver overwrites it, then the first writes it back. Two letters fighting over one envelope, each self-check finding the other’s contents. Both loop on "not yet propagated" — and neither is a propagation problem.

The fix is to stop asking for both: request the wildcard only, and drop the apex SAN unless you serve the bare domain. This one ate a real evening on robin before the pattern was obvious.

A permission failure in a propagation costume

The third trap doesn’t even mention permissions. A Cloudflare API token with Zone:DNS:Edit alone looks sufficient — it can edit records, what more could a DNS solver want? But cert-manager first has to enumerate zones to find the zone ID, and that needs Zone:Zone:Read. Without it, the failure surfaces not as 403 but as — you guessed it — "not yet propagated".

Add Zone:Read to the token at Cloudflare and it unblocks instantly. The token string doesn’t change, so no secret rotates; the fix never touches the cluster at all.

Three different root causes. One identical symptom. That’s the real menace of this error message: it names the last observation (“I can’t see the record”), not the reason.

The principle

A self-check is only worth what its vantage point is worth. If your verifier stands outside — a CA on the public internet, a monitor beyond the LAN, an auditor reading the published copy — then a check that observes from inside isn’t a weaker version of the truth. It’s a different truth, and it can disagree indefinitely.

So when a system checks its own work before presenting it, ask one question first: does it look from where the judge will look? Check the postbox the postman collects from — not the mail slot in your own hallway.

Bit the media-gateway wildcard on elfastc, 2026-07-02 — cert-manager v1.19.5 issuing *.bztmon.org via Cloudflare DNS-01, on a fleet where the internal Pi-hole owns the zone and public resolvers hold the proof.