One way in, and a dead route looks like a dead estate

This is my homelab: single-node Kubernetes clusters on Talos with Cilium as the CNI, one operator, remote access over a mesh VPN whose only way onto the LAN is a subnet router. Nothing here describes an employer's systems. Both incidents below are dated, the second one belongs to that particular stack, and the design rule at the end is what I run now.

"Git is down." It was not. The server was serving, its access log showed requests being answered, and a reference listing from another machine came back while the diagnosis was under way. What had failed was the path in.

From the reporting chair those two situations are identical. Nothing resolves, nothing connects, every service is equally unreachable. The evidence available to the person experiencing it cannot separate "the estate is down" from "my way in is". When there is one gate, a stuck gate looks like a demolished building.

Prove the service from a chair that is not yours

The order that resolves this quickly inverts the instinct. Do not open the service's logs first. Establish that somebody else can reach it, from a position that does not share your path.

# run from a machine already inside the network, not the one reporting the outage (lab example address)
GIT_TERMINAL_PROMPT=0 timeout 10 git ls-remote --exit-code https://git.example.org/lab/site.git refs/heads/main >/dev/null \
  && echo "the server advertised refs/heads/main from inside - now make the same request from the reporting chair"
# a reference listing over HTTPS: the server must answer it, but no objects move; --exit-code turns an absent ref into exit 2 rather than a silent empty success

A request the service itself has to answer, made from a chair that does not share your path, is the discriminator. A bare TCP connect is cheaper, but where one ingress fronts many services it only proves the ingress is listening, and ICMP is filtered on this LAN, so ping proves less than that. Be honest about what the listing proves, though. It is not a fetch, and a success inside does not acquit the service of everything: a deny that applies only to your source, split DNS giving the two chairs different answers, IPv4 on one side and IPv6 on the other, a second backend behind the same ingress, or a fault that comes and goes all survive it. What it does is narrow the fault domain, and the narrowing only holds if the two chairs make the same request - same hostname, same resolved address, same protocol, same operation. If the inside request succeeds and the outside one fails, the next minute belongs to the path, not the server.

The complaint above, on 4 August 2026, resolved exactly that way. The service's own log proved it was serving, a listing from another machine confirmed it, and the mesh client on the reporting workstation turned out to have stopped. A client restart fixed it. Nothing in the estate had changed, and the estate had "been down" for as long as it took to work through that list in the right order.

Name the node holding the route

In Tailscale, when several nodes advertise the same subnet, one is elected primary for a given client and carries its traffic. In my configuration, with one subnet and default failover, that makes the winner the node carrying every remote connection into the network, and the first suspect. The product can route differently per region and per client, so check what yours does before reading one winner as a universal shape.

set -o pipefail
timeout 10 tailscale status --json \
  | jq -re '[.Self] + [(.Peer // {})[]]
            | map(select((.PrimaryRoutes // []) | length > 0))
            | if length == 0 then error("no node reports a primary route: unknown, not clean") else .[] | "\(.HostName) \(.PrimaryRoutes | join(","))" end'
# route-owner metadata from the local daemon (tailscale 1.102.2, read 8 October 2026): one line per node carrying a subnet.
# it names the holder; it says nothing about whether packets get through. empty is an error, not a clean bill.

The failure I did not see coming, on 29 July 2026, is what this looks like when the holder is the wrong kind of host for this stack. I had subnet routers running as pods inside two of the clusters, which the operator supports and which is tidy. The router that won the election was a pod on Talos, where iptables failed with exit status 3, so the mesh daemon logged that it could not install a firewall and carried on. In its default mode the daemon rewrites the source address of forwarded packets with that firewall; with no firewall, the rewrite was never set up. Two records survive from the diagnosis: that boot-log line, and a drop counter on the source node with reason Invalid source ip. The path in between is my reading of those two. Packets to a service on another cluster left the node and were rewritten there on the way out, so they worked; packets to a service on the same cluster had the service address translated at the pod's own interface, kept their raw overlay source, and the CNI's source check dropped them before they left it. Nothing at the destination saw a packet, so the flow logs at the destination showed nothing. I did not capture packets to confirm the rewrite points, and I am not going to claim I did.

So whichever cluster won the route election went dark from the mesh while its neighbours stayed fine, and the symptom read as a broken service on that cluster rather than a broken route. The diagnosis chased DNS, then a network policy, then the application. DNS and the application were innocent. The policy had a real gap of its own, fixed on the way, but it was not this fault. A pod emitting packets with an address it does not own is a missing rewrite, not a policy decision, and the fault class actively disguises itself as the thing you are about to go and debug.

The lesson is scoped to that stack. It does not say a router in a pod is always wrong, or that every subnet router must rewrite addresses: the product supports site-to-site routing with the rewrite off and return routes in place, and that is a legitimate design. What I can say is that a router in a pod on this OS with this CNI could not do the rewrite it was configured to do, and failed in a corner that pointed away from itself. My way out was to remove the dependency. The gate is no longer a tenant of the building it lets you into. The subnet is advertised from a plain Linux host that is not a tenant of any cluster it routes to, with a working firewall and no hairpin case. The in-cluster routers are deleted from the repositories, with a README explaining why, so a sync cannot bring them back.

Advertise, never accept your own subnet

There is a second way to jam the gate, and it is quieter. A node that lives inside the subnet it advertises must not accept the overlay's route for that same subnet, or on authentication it will take the mesh's version of its own LAN and send local replies into the tunnel. That is the narrow rule: your own overlapping prefix. A site that accepts routes to other remote subnets is doing something different and legitimate. I did the narrow mistake to the gate itself once; the shape of it, and the out-of-band recovery, is told in The bastion is a pattern, not a box. On this host tailscale debug prefs shows RouteAll as false, which is the configuration half of the proof. The other half is the kernel: ip route get for a LAN address should name the LAN interface and not the tunnel, and a connect to a LAN service from that host should succeed without the mesh. A preference is not a path.

Suspect the gate before the building

What changed after these two was not the topology, although the topology did change. It was the reflex. A report that everything is down now gets read as a claim about the path first. The path gets walked link by link from a chair that is not the one complaining, and the route holder is named before any service log is opened.

Redundancy, when it comes, is a second host of the same kind as the gate, and the test of whether it counts is a list of shared failures: power, the host, the switch it hangs off, the DNS it resolves with, the control plane that admits the client. Two hosts that share three of those are one path with a spare chassis. A second path also does nothing for a stopped client daemon on the reporting machine, which is the fault I actually had on 4 August. Compare paths by what they do not share, and keep the list honest.

With one gate in, "unreachable" describes the gate until something proves it describes the building.

Live in the lab: the LAN is advertised by a host that sits outside every cluster it routes to, which removes the fault that made one cluster look broken when the route was to blame. It does not stop a dead route looking like a dead estate.