The best migrations delete more than they install
Ask why the fleet ran flannel and there was never an answer worth writing down. Nobody chose it. Talos bundles flannel, so flannel ran. Kubernetes ships kube-proxy, so kube-proxy ran. LoadBalancer services need something on the LAN to answer for their addresses, so MetalLB moved in. HTTP needed terminating, so ingress-nginx went up, and later Traefik replaced it - on one cluster, Traefik still answers on ingress-nginx's old IP, the fossil of a hop nobody remembers deciding.
That's how most platform stacks are built. Not designed - accreted, the way an old house grows. A lean-to here, a carport there, each extension the quickest answer to the question of the day. And every extension arrives with its own roofline: a version to pin, a changelog to read, failure modes to learn, alerts to write.
This week the fleet finished landing on Cilium, and the honest record of that landing is the point of this post. It wasn't a product evaluation. It wasn't an adoption. It was a demolition schedule, and the stack left standing is smaller than the one we started with.
Count the rooflines, not the features
The starting shape, across a fleet of single-node Talos clusters: flannel carrying pod traffic, kube-proxy rewriting every service address, MetalLB answering ARP for the LoadBalancer VIPs, Traefik terminating HTTP. Plus a set of network policies that were authored, committed and completely inert - flannel can't enforce them, so the plans existed and nothing was ever built to them.
Four separate projects for four concerns, and a fifth concern going unmet. Each project a chart or a DaemonSet with a version, a CVE feed, an upgrade cadence, and its own distinct way of ruining an evening. None of them wrong individually. All of them defaults, and a default is a decision someone else made, for a fleet they've never seen.
What started the demolition wasn't "Cilium is better than flannel". It was noticing that one engine could carry all five concerns - the pod network, the service path, the VIP announcements, the policy enforcement, the flow visibility, and that one cluster in the fleet already ran that engine, a quiet standing proof. Four rooflines could become one.
Swap the frame under an occupied house
Stage one, June: replace the CNI in place, on live clusters, no rebuilds. Per cluster the sequence is short. Set cni: none in the machine configuration, one reboot, install Cilium 1.19 with a pinned set of values, delete the flannel DaemonSet, then restart every non-hostNetwork pod so it leaves flannel's address management for Cilium's. Order the clusters lowest blast radius first; the control-plane hub goes last, once the procedure has become boring.
Two disciplines mattered more than the sequence. First, confirm the node rebooted at all - the apply returns immediately and the API often answers before the machine has cycled, so trust the uptime counter, not reachability. Second, those inert policies stop being inert the moment Cilium arrives: lines on a plan under flannel, enforced at the instant of cutover. Audit every allow-list before migrating the cluster that holds it.
Four live clusters converted in a day. Zero rebuilds, zero data loss, and the fleet verified green afterwards - 98 of 98 GitOps applications healthy. Then stage two, the reason enforcement mattered at all: a default-deny floor in every namespace, rolled fleet-wide with the observe-first method that has its own post - watch the real flows, then deny everything else (the why lives here).
Then pull the extensions down
Stage three, July - finished this week. With Cilium carrying pod traffic everywhere, the remaining structures came down one by one.
kube-proxy went first. Cilium's eBPF datapath took over service routing entirely, reaching the Kubernetes API through Talos's KubePrism endpoint on localhost:7445, after which the kube-proxy DaemonSet was deleted outright. Then MetalLB: Cilium's LB-IPAM handed out the same VIPs from the same pools, L2 announcements answered the same ARP queries, and every service kept its address through the swap. One reboot per node. On the media node, Cilium's Gateway API absorbed the ingress role as well - one more structure gone. MetalLB is now extinct across the fleet. Not deprecated, not legacy-mode: extinct.
The last demolition was managerial. The CNI no longer belongs to Helm at all - it ships as a Talos inline manifest, which means the network lives inside the machine's own configuration document. A rebuilt node comes up with its network the way it comes up with its kernel: because the document says so. Zero Helm release secrets remain anywhere in the fleet.
Two structures fought back
There is always a price, and this one came due twice.
The first trap is the nastiest kind of failure: the change that succeeds and does nothing. Deleting Helm's release record orphans the resources, but it does not release Helm's per-field ownership inside them. So the flip to the eBPF datapath applied cleanly, the node rebooted on schedule, and KubeProxyReplacement still read False, because a config field Helm still owned silently refused the new manager's apply. No error. Nothing in the diff. It fired on every single cluster we flipped, so it's now a standard step rather than a contingency: steal the whole manifest's ownership server-side - ```
kubectl apply --server-side --force-conflicts --field-manager=talos -f cilium-rendered.yaml
The second was musical chairs with the VIPs. Mid-swap on one cluster, a service without a pinned address grabbed another service's freed IP the instant the old announcer released it - the LLM backend ended up sitting on the ingress controller's address. The fix is an ordering rule: pins first. Land the address pins while the old announcer is still alive, confirm they hold, and only then retire it. Two commits, in that order, every time.
## Count what you removed, not what you added
Consolidation is usually sold on the new tool's feature list. That's the wrong ledger. The value of landing on one engine wasn't what Cilium added - it was what the landing let us remove: a CNI, a service proxy, a VIP announcer, and on one node an ingress controller, each of which had been a version to pin, a feed to watch, an alert to tune, a way to be paged. Policy enforcement and flow observability never needed projects of their own; they came up with the same engine, side effects of the consolidation rather than line items on it.
Every structure you tear down is one that can never fall on you. Judge the next migration by its demolition list.
*Live across six single-node Talos clusters: Cilium 1.19.4 shipped as a Talos inline manifest, the eBPF datapath where kube-proxy used to be, LB-IPAM answering every VIP MetalLB once held, and Hubble watching the flows on every cluster.*