~/jwright
← All posts
·5 min read

SNAT ate my source IP

A glowing envelope passing through a dark sorting machine that stamps over its return address

Here’s the uncomfortable default: give a Kubernetes service a LoadBalancer IP, leave externalTrafficPolicy at its default of Cluster, and every packet that arrives gets its source address rewritten to the node’s own. Not by an attacker. By the load balancer, on purpose, as routine plumbing.

I found out the way you always find out — by writing a network policy for it. LAN clients live on 10.0.11.0/24, so I wrote a fromCIDR 10.0.11.0/24 allow. Applied cleanly, showed Valid=True, dropped every real user. Hubble told me why: the traffic wasn’t arriving from the LAN at all. Cilium tagged the source world — the identity of anywhere, the one you can’t restrict without restricting everything.

It’s a letter that reaches you carrying the sorting office’s return address instead of the sender’s. The sorting office isn’t lying to you; it’s how the machinery forwards mail. But you can’t write a “letters from Alice only” rule when every envelope on your doormat says it came from the depot.

SNAT rewriting the client source before policy evaluation, and ETP:Local preserving it

Why the depot stamps over the sender

The rewrite has a reason. With externalTrafficPolicy: Cluster, any node can accept traffic for the service and forward it to a backend pod on a different node. For the reply to route back through the node that forwarded it, that node SNATs the connection to itself. Balanced spreading, bought by destroying the source.

The consequence lands exactly where you can’t see it coming: policies bind to what arrives, not to what was sent. The client’s packet left home with an honest return address; the depot stamped over it in transit; your policy — evaluated after the rewrite — never meets the client at all. There is no rule you can write for an identity the network erased one hop earlier.

Preserve the sender, then commit it

The fix is one line: externalTrafficPolicy: Local. Only nodes running a backend pod accept the traffic, no cross-node forwarding happens, no SNAT is needed — the client’s real address survives to policy evaluation. Now fromCIDR 10.0.11.0/24 matches LAN clients, and a cross-cluster peer shows up as its node IP — the peer node, not the peer pod, because separate clusters masquerade pod traffic on the way out — pin-downable with a /32. That’s how the LLM gateway’s ingress rule got to be a single line: one legitimate consumer, the agent cluster next door, exactly one /32 allowed in.

Here’s the one that costs a 2am. I made that change with kubectl patch — quick, out-of-band, worked immediately. The service was GitOps-managed with self-heal on. Within seconds ArgoCD noticed the live object differed from git and put it back the way the repo said, which re-enabled the SNAT, which re-broke the policy, which took DNS down with it. The change must be committed, or the platform will politely undo your fix while you sleep.

One caveat that keeps it honest: some paths still rewrite. A client arriving through a tailnet subnet router gets masqueraded at the routing node even with Local set — it lands as world regardless. Some envelopes pass through a second depot you don’t control.

When the depot closes, every envelope changes

This week the fleet finished swapping kube-proxy for Cilium’s eBPF replacement. Same services, same addresses, same policies — and a traefik that had answered LAN probes for months went dark, under a policy nobody touched.

It was never supposed to answer. That traefik fronts zero routes, and the tightest floor guards it — a policy admitting host, because kubelet probes need it, and nothing else. But kube-proxy’s SNAT had been dressing LAN probes as host — a different depot, a different stamp, the same disease. The replacement closed the depot. Every envelope arrived carrying its real return address, the probes landed as world, and the policy dropped them — exactly as written. Nothing broke. The policy finally saw the truth, and the truth matched the original intent: answer no one.

But if you didn’t know why it used to answer, this is the morning you file an outage ticket and “fix” a policy that was never wrong. The rewriting machinery is itself a moving part. Change the plumbing and every identity changes with it — which is why only observed flows stay true, and remembered ones quietly expire.

Read the envelope before you write the rule

The real lesson isn’t the one-line fix. It’s that I wrote a policy for the traffic I imagined — clients on the LAN, arriving as themselves — instead of the traffic that arrived. The packet’s story gets rewritten at every hop: SNAT here, DNAT there, a masquerade at a routing boundary. The sender’s truth and the receiver’s truth are different documents.

So the discipline, before any allow rule exists: observe first. Open Hubble, watch the real flows, and note the identity the traffic actually carries when it reaches the endpoint — world, host, cluster, a pod label, a CIDR. Then write the rule for that. A policy written from the architecture diagram is a guess; a policy written from observed flows is a fact.

The principle

A network policy is a doorman checking return addresses, and the postal system rewrites return addresses as a matter of course. You don’t secure what was sent — nobody ever sees what was sent. You secure what arrives. Go and look at the envelope first.

Bitten and fixed on a six-cluster Talos fleet running Cilium 1.19 — kube-proxy replaced by the eBPF datapath fleet-wide, MetalLB retired, and a standing rule: Hubble before policy, every time.