~/jwright
← All posts
·5 min read

Observe first, deny second

A dark reserve at night, instrument beams tracing animal paths before any fence exists

Here’s the uncomfortable default: when it’s time to lock a namespace down, almost everyone opens the architecture diagram and starts writing allow rules from it. This talks to that, that talks to the database, done. The diagram is confident, tidy — and wrong. It omits the probe traffic, the DNS hop, the controller’s back-channel to the API server, the one call some library makes that nobody documented.

Policy written from a diagram feels like engineering. It’s fiction with enforcement attached — and enforcement doesn’t care that the fiction was well-intentioned. It drops the flow anyway, in production, at whatever hour the flow next happens.

I made the case for the default-deny floor itself in the trust-nothing post; this is the other half: how you roll it across a live fleet, namespace by namespace, without an outage. The inversion is simple. Don’t fence first and see what starves. Survey the reserve, then build the fence.

The observe → deny → verify → commit loop

Survey before you fence

A ranger doesn’t fence a reserve from a map. They track the actual animals for a season — where they drink, which paths they take at night — because the fence has to leave every real path open and close everything else.

Hubble is that season of tracking. Before a single policy exists, watch the namespace’s real flows — every connection, source and destination identity, port. That record is the allowlist. Not the docs, not your memory of the architecture, not the diagram. Every rule you write should point at a flow you watched happen. Start with the lowest-risk namespaces and leave the control plane for last: mis-fence a leaf app and one thing breaks; mis-fence the plane that runs deploys and everything does.

Build the fence where you can tear it down

Apply the policy trio — the default-deny plus its allows — out-of-band with kubectl, not through git. This is deliberate, and it’s the step people skip. The GitOps reconciler doesn’t know the policy exists, so it can’t fight you over it, and rollback is one kubectl delete — instant, no commit, no sync wait. Push the policy through git first and you’ve inverted your own safety: selfHeal means the reconciler restores whatever you delete, in seconds, while you stand there believing you rolled back.

Out-of-band is a temporary state, not a destination. It’s the fence held up with clamps while you check the gates.

Prove it bites, prove nothing bleeds

Two verifications, and both are mandatory because each one lies without the other.

First, prove the policy enforces at all. On Cilium 1.19 a policy can be accepted but inert — it passes a server-side dry-run, sits in the cluster looking correct, and enforces nothing. A policy that looks live and isn’t is worse than no policy: it changes what you believe without changing what the network does. Read the live object’s status conditions and demand Valid=True, then watch a forbidden connection get dropped. “It applied cleanly” and “it’s enforcing” are different sentences.

Second, prove nothing legitimate is bleeding. Back to Hubble — zero drops on real flows, and the real consumer path exercised end-to-end, not just a curl from your own shell.

Only when both hold does the policy go to git. The reconciler adopts it, out-of-band ends, and the rollback story changes shape: from here, undo means git revert — because selfHeal now defends the policy as hard as it would have fought your rollback.

The sternest test so far wasn’t a rollout at all: the fleet’s datapath was swapped out underneath the policies — kube-proxy replaced by Cilium’s eBPF kube-proxy-replacement on the last three clusters, one reboot each — and afterwards every namespace’s floor re-verified enforcing: 30/30, 32/32, 27/27 Valid=True, zero legitimate drops. The fence held while the ground under it was replaced.

The paths nobody draws

Three flows exist in every namespace and appear on no diagram. Omit any one and you pay:

  • DNS — allowed by label, never by IP. Pin the resolver’s ClusterIP into a rule and it never matches at all — the address is rewritten to a backend pod before policy judges the flow. Select k8s-app: kube-dns and it holds forever.
  • The kubelet’s probes. Health checks arrive from the host entity on every workload. Forget the allow and your pods go NotReady the moment the deny lands — the fence starving the animals it was meant to protect.
  • API-server egress. Every controller and operator talks to the Kubernetes API constantly. A controller that “manages X” usually manages it through the API server, so this one allow often covers its whole lifecycle.

The principle

The method generalises past network policy. Any control that turns assumption into enforcement — firewall, RBAC, admission — is only as honest as the evidence it was written from. Observation first, out-of-band while unproven, permanent only after it’s earned it.

Policy written from observation is engineering. Policy written from a diagram is fiction — and production is where fiction gets fact-checked.

Rolled out across all six Talos clusters on Cilium 1.19 — every namespace surveyed with Hubble before its deny landed, the control plane last, zero outages; the floor re-verified enforcing after this week’s kube-proxy→eBPF datapath swap.