A healthy deployment is not a reachable one

This is my homelab: single-node Kubernetes clusters, one operator, and nothing here describes an employer's systems. On the hub the gateway is Traefik v3.6.7 acting as a Gateway API gateway, routes are HTTPRoute objects, network policy is CiliumNetworkPolicy on Cilium v1.19.4, and Argo CD v3.4.7 applies everything from git.

On 2026-09-02 I onboarded a small read-only web application. The rollout finished, the HTTPRoute was accepted, the endpoints came up Ready and the pod answered its own health checks. Every request from outside sat until it timed out. What was missing was not in the application's manifests. The gateway sits behind a default-deny policy with an explicit list of the backend namespaces it may open connections to, and the new namespace was not in it. The single sign-on round trip at the edge had completed before the blank page, so the request reached the gateway and died on the way out of it. I am withholding the application's name; nothing else about the incident is generalised.

A tenant can finish a fit-out to the last light switch and still find that the building's security desk never added them to the door list. That is the whole shape of it.

Reachability is a path, not a status

Readiness is a property of one pod: its own probe passed. Application health is what the process says about itself. Routing is whether the gateway holds a rule that names the backend. Network policy is whether the packets are allowed at each point they are checked. Reachability is a request from a particular client succeeding over that particular path, and only the whole path proves it. Every layer here was green except the one nobody had asked.

Where the packet was checked

Both ends of a gateway-to-backend connection are isolated by default-deny in this lab, so one new connection needs two allowances: the gateway's egress towards the backend namespace, and the backend namespace's ingress from the gateway on the application port. Reply traffic on an allowed connection is permitted statefully, so the backend needs no general egress rule to answer. In this incident the missing piece was the first one, and Cilium enforced it as the gateway's egress. The application's own outbound traffic, DNS and whatever it calls, is a separate connection with its own rules. That is the third entry in my onboarding checklist, and it is not part of this connection.

The gateway's rule is a CiliumNetworkPolicy named traefik-allow in the traefik namespace. Its endpoint selector is the Traefik pods, and this excerpt sits under spec.egress. The destination is any pod in a listed namespace, on any port: that entry carries no toPorts on purpose, because the port restriction lives in each destination namespace's own ingress rule.

# CiliumNetworkPolicy traefik-allow, namespace traefik, under spec.egress.
# endpointSelector: app.kubernetes.io/name: traefik. Two of the listed namespaces shown.
    - toEndpoints:
        - matchLabels:
            io.kubernetes.pod.namespace: argocd
        - matchLabels:
            io.kubernetes.pod.namespace: gitea

The other half is in the application's namespace: a default-deny policy selecting every pod, and an allow policy whose ingress rule names the gateway's namespace and the application's port. Both halves live in the platform repository.

What the failure looked like, and what it did not

A policy drop discards packets. What a person sees depends on the layer above. Traefik keeps trying to open the backend connection until its own dial timeout, then answers the client; in my case the browser showed a spinning page. Traefik's access log is not enabled in this lab, so nothing was recorded at the proxy either; with it on, the gateway's timeout status would have been there. The backend logged nothing because no request ever arrived. None of that is a rule about every failure. It is what this one looked like.

Ask the datapath the right question

Hubble records each flow with its verdict. Filtering on the gateway's namespace and DROPPED alone does not name the culprit. Correlate one request with its time, both endpoints, the port, the direction it was enforced in, and the reason:

# Read-only. On this single-node cluster the agent's view is the whole cluster; on a multi-node
# cluster query Hubble Relay (kube-system/hubble-relay) with the hubble CLI instead.
CTX="admin@cave"
APP_NS="learn"
kubectl --context "$CTX" -n kube-system exec ds/cilium -c cilium-agent -- \
  hubble observe --since 5m --verdict DROPPED --from-namespace traefik --to-namespace "$APP_NS" -o json \
  | jq -r '(.flow // .) | [.time, (.source.namespace // "-"), (.destination.namespace // "-"),
                           .l4.TCP.destination_port, .traffic_direction, (.drop_reason_desc // "-")] | @tsv'

traffic_direction says whether the drop was enforced as the source's egress or the destination's ingress; a drop that originates at the gateway can still be enforced at the backend's ingress. drop_reason_desc separates a policy denial from everything else. Naming the exact policy means reading what is imported on that endpoint in that direction. The default --last 20 window and the buffer behind it can miss the request you are chasing, so filter by time and repeat the request while watching. No matching output does not prove the traffic is allowed; it proves only that no drop was recorded in that window.

Prove the path, not the pod

# Read-only, from the client's own location. TLS is verified, a redirect is not accepted as
# success, and the body has to contain the page it should.
curl --fail --silent --show-error --max-time 10 -o /tmp/planes.html \
  -w 'status=%{http_code} redirect=%{redirect_url}\n' https://learn.bztmon.com/planes
grep -c '<title>Two delivery pipelines' /tmp/planes.html

A login redirect or a gateway error page returns a status too; only the expected page counts. Compare that with the same request made from inside the cluster against the service, and with the pod's own probe, and the three answers locate the break. A pass covers that path at that time. It does not prove every route, identity or client network works.

Who owns what, here

Two repositories, one person. The application repository holds the code and the build; its pipeline pushes an image and opens a digest-bump change into the platform repository. The platform repository holds the Deployment, the HTTPRoute and both network policies, and Argo CD applies it. I am the committer on both, so the review is mine on both, and there is no second reviewer.

Argo CD can constrain what a source may deploy through AppProject restrictions, and the credentials it holds and the cluster's admission rules constrain what it can apply. In this lab the one project has no source, destination or resource restrictions, and the controller's role is unrestricted, so the boundary today is who can write to the platform repository, not an enforced control. Two repositories are not a security boundary by themselves. Cilium allow rules are additive: a second policy selecting the same pods widens what is allowed, so anyone who can commit a policy into that namespace can widen it. That is the arrangement, stated plainly.

The chapter this post condenses describes a multi-team version with separate reviewers and project restrictions. That is an illustrative operating model, not what runs here. The onboarding contract I use is this lab's: an application package, and one reviewed platform change covering the namespace, the two policies, the route and whatever platform services the application needs. Many services reuse what already exists and need no new storage, secret or registry credential.

The negative test I would run, and have not: a change in the application repository that tries to reach beyond its namespace should be stopped by the controls that are actually configured. Until it is run, containment is a design intent, not a result.

Observed 2026-09-02 in this lab on Traefik v3.6.7, Cilium v1.19.4 and Argo CD v3.4.7. The full dossier is chapter 03 on learn.bztmon.com. Revised 2026-09-06.