A healthy tunnel can still break cluster DNS

This is my homelab: single-node Kubernetes clusters, one operator, and nothing here describes an employer's systems. The acquisition stack runs in one namespace, and the clients that should only ever talk through the VPN share a pod with the tunnel client. Sharing the pod gives them its network namespace. What decides their egress is the routing and firewall the tunnel container configures, and that carries deliberate exceptions for cluster-internal addresses; the pod boundary on its own guarantees nothing about where a packet goes.

The symptom, the first time, was an application that had stopped working. The download client sat idle, logging that it could not resolve its server. Read that line and the natural next step is the client: its configuration, its server list, its credentials.

That would have been an hour spent on the last domino in the line. The client was fine, and so was its configuration. What had fallen first was several places back, and everything in between had faithfully reported being knocked over.

Ask for a name the app does not care about

The discipline that gets there fast is refusing to investigate the reported component first. Ask what it depends on, and check the dependency before the thing itself. Can it resolve a name at all, not its server, any name?

# Read-only. Two questions: can this container resolve anything, and which resolver is it reading.
# The names are the ones in your own manifests; every call is bounded so a dead resolver cannot
# hang the check.
NS="media"
POD="deploy/vpn-pod"
CLIENT="downloader"
timeout 15 kubectl -n "$NS" exec "$POD" -c "$CLIENT" -- nslookup example.com
timeout 15 kubectl -n "$NS" exec "$POD" -c "$CLIENT" -- cat /etc/resolv.conf

If a name with nothing to do with the application fails as well, the server list is not where to look next; something shared is. It does not prove the application has no fault of its own, only that the shared path fails first. The second line names the resolver, and in this stack that resolver belongs to something else entirely.

One file, read by every container

Containers in a pod share a network namespace: the same interfaces, the same routes, the same exit. They do not share a filesystem, so the reason /etc/resolv.conf reads the same in every container of this pod is not the namespace. The kubelet builds one DNS configuration per pod and hands it to the container runtime, and on this cluster's runtime that is one file in the pod's sandbox directory, bind-mounted into each container. I checked that on 2026-09-18 by reading the mount table from inside three containers of the pod: the same source path, the same device and inode. So a container that can write to it rewrites it for all of them.

The tunnel container does exactly that at startup. It replaces the cluster nameserver with its own DNS server on localhost, which forwards over the tunnel to a public resolver, so that no name lookup escapes the tunnel either. The image has a switch to keep the existing nameserver, DNS_KEEP_NAMESERVER, and it is off by default. That is how the rewrite arrives in a pod nobody configured for it. The search domains the kubelet wrote survive it; only the nameserver line changes, and every container I opened was reading the changed one.

# Read-only. The address this container's traffic leaves from, seen from outside.
timeout 15 kubectl -n "$NS" exec "$POD" -c "$CLIENT" -- wget -qO- --timeout=8 https://api.ipify.org

Working, that prints the tunnel's exit address rather than the house's. That is one request at one moment leaving by the tunnel; it says nothing about what happens while the tunnel is down, about other protocols, or about the exceptions the tunnel's firewall carries. The guarantee, such as it is, comes from that firewall and its configured exceptions, not from the pod. Failing, the probe fails for the same reason every other name did, which still puts the tunnel upstream of everything the application does.

The second time, the tunnel was fine

The first episode was a tunnel that had dropped, taking its resolver with it, and the client came back the moment the tunnel did. The second episode, on 2026-09-16, was quieter and taught me the mechanism above. The indexer manager in the same pod reported that every application it pushes configuration to had been unavailable for more than six hours. The tunnel was up. Public names resolved. What did not resolve was anything ending in svc.cluster.local, because the resolver every container in that pod reads is the tunnel's, and a public resolver has never heard of my cluster's names.

# Read-only. The same URL and Host header twice: once resolved by the pod's resolver, once pinned
# to the Service's ClusterIP with --resolve, so the only variable is who answers the name. The
# address comes from the API, never hard-coded.
INDEXER="indexer"
APP="radarr"
PORT="7878"
IP="$(kubectl -n "$NS" get svc "$APP" -o jsonpath='{.spec.clusterIP}')"
[ -n "$IP" ] || { echo "no ClusterIP for $APP" >&2; exit 1; }
URL="http://$APP.$NS.svc.cluster.local:$PORT/ping"
timeout 20 kubectl -n "$NS" exec "$POD" -c "$INDEXER" -- \
  curl -s -o /dev/null -w 'by name: %{http_code} %{errormsg}\n' --max-time 8 "$URL" || echo "by name: exit $?"
timeout 20 kubectl -n "$NS" exec "$POD" -c "$INDEXER" -- \
  curl -s -o /dev/null -w 'by address: %{http_code}\n' --max-time 8 --resolve "$APP.$NS.svc.cluster.local:$PORT:$IP" "$URL" || echo "by address: exit $?"
# by name: 000 Could not resolve host / by address: 200. The path is open; the name is dead.

That pair of lines is the diagnosis. Reaching the same service by its address shows that this TCP path was open and allowed at that moment, so what is broken is resolution, and from the tunnel's point of view it is broken by design. The blast radius was narrower than the alarm: the indexer manager only pushes indexer configuration to the other applications, and the configuration they already held kept working, so searches ran that day while changes stopped propagating. Where those cluster names should be answered inside that pod is a design decision, and not one this post makes. It was not the download client's fault either time.

Read the log of the thing that broke

Every layer above the fault describes the fault it experienced, accurately and uselessly. The layer that actually broke usually says so in plain language, and nobody is looking at it:

# Read-only. The tunnel container's own account of its DNS server and its exit. A grep is triage,
# not evidence; it misses whatever the log calls by another name.
TUNNEL="tunnel"
kubectl -n "$NS" logs "$POD" -c "$TUNNEL" --tail=200 | grep -iE 'dns|resolv|exit'

In the first episode the tunnel reported its own failure and ended the investigation. In the second it reported success: its DNS server listening, and the public resolver it forwards to. Both were the truth. Neither was the client's problem, and neither was written where I was looking.

Test the dependency, not the alarm

The client came back the first time with its configuration untouched. The second time the client was never broken at all; the sync it depends on was. The component that raised the error needed nothing, and the component that needed everything raised no error I was looking at.

The method that found both is the same, and it is not "check DNS". It is: draw the dependency graph of the thing that complained, test each dependency with a probe that does not care about the application, and let the fault be wherever the first failing probe says it is. Twice that was the resolver. Next time it will be something else, and the method will still find it.

An error message tells you where a failure surfaced. It has no opinion about where it began. Read the line backwards.

Observed in this lab: the first episode in 2026-07 on the previous host, the second on 2026-09-16 on gluetun v3.41.1 inside a shared pod on Talos; the shared resolver file was confirmed by mount identity on 2026-09-18 (Talos v1.13.9, containerd 2.2.7, kubelet v1.35.8). The clients still share the tunnel's pod. What keeps their traffic inside the tunnel is its firewall, not the pod.