← All posts

My servers don't have SSH, and that's the feature

A sealed obsidian machine with a single structured API port of light

The first thing everyone asks about Talos Linux: how do you get in? You don’t. There is no SSH daemon to connect to, no shell waiting behind it, no package manager if you got there, and no login even at the physical console. The operating system boots, runs Kubernetes, and answers exactly one thing: a mutual-TLS gRPC API. Six machines in this fleet, and not one of them is a place I can visit.

That sounds like a limitation you’d tolerate for the security. It took about a week of running it to realise it’s the point.

A server you can’t visit is a server you can’t drift

Every hand-run command on a traditional box is an unrecorded change: a config edited to test something, a package installed during an incident, a daemon restarted with a flag nobody wrote down. None of it is in git. All of it is load-bearing by next month. Configuration drift isn’t a failure of discipline — it’s the inevitable product of machines that accept hands.

Talos removes the hands. The entire machine — disks, network interfaces pinned by MAC, kernel modules, registry mirrors, even the manifests the cluster boots with — is one declarative document, applied through the API, versioned in git. There’s no side door through which an undocumented change can arrive, because there’s no door. The machine config isn’t documentation of the machine. It is the machine.

Many doors and drift, versus one API and a document

Operating through the keyhole

Day-two work changes shape. Logs, service status, process lists, even packet capture arrive through typed API calls, not a terminal session — and every mutation is an apply: edit the document, run a dry-run diff that says precisely what will change and whether it costs a reboot, then commit it. The dry-run gate is the quiet superpower — on a machine you can’t shell into, “what exactly will this do?” stops being a guess and becomes an answer the API owes you before anything moves.

This week put that shape under load. Three of the six nodes had their network datapath swapped — kube-proxy out, Cilium’s eBPF replacement in — and their registry credentials rotated, one reboot each, entirely through the API. Nobody logged into anything, because there is nothing to log into. The credentials never touched a live machine either: they’re substituted into the document at render time from an encrypted file and arrive as configuration, not keystrokes — there is no terminal to mistype them into.

Two habits from that work show what keyhole operation feels like. After an apply that costs a reboot, the API answers again before the machine has cycled — the door reopening proves nothing. So you read /proc/uptime through the API and demand a number seconds old; the check has caught a node still showing 413,223 seconds of uptime with the apply long returned. You don’t trust the door. You read the building’s own clock through the keyhole. And before calling a node good, you have it pull a container image through that same API — the machine exercises its new registry credential end-to-end while you watch from outside.

When something’s genuinely wrong, the recovery isn’t archaeology on a mutated filesystem — it’s reconciliation: re-apply the known-good document, or in the worst case rebuild the node from it in minutes. The machine has no state worth rescuing because nothing was ever hand-placed on it. Pets die of unknown illnesses; documents get reprinted.

The honest trade

The console is gone as a control surface, not as a truth surface — and that distinction still bites. A node once dropped off the network in a way that looked, from every remote signal, like a total lockup. The physical screen told a different story: the box was alive and healthy, only its network path had wedged. The lesson isn’t “you need a shell” — a shell would have shown the same thing more slowly. It’s that firsthand evidence still outranks remote inference, and an API-only fleet needs its operators to remember the difference on the day it matters. The same lesson came back this week when a node went dark at layer 2: the fix was a screwdriver, not a session — no SSH daemon answers on a dead NIC either.

The principle

Control isn’t the ability to touch a machine — it’s the guarantee that the machine matches a document you trust. Every interactive door a server offers is a place where reality and the record can quietly diverge, and everything that makes fleets miserable lives in that gap. Seal the doors, apply the document, keep the diff. The machine you can’t log into is the only one whose state you truly know.

Live across a six-node Talos fleet: machine configs generated and versioned in git, applied over mTLS with dry-run gates, nodes rebuilt from documents when hardware moves. Three of the six had their network datapath swapped and their registry credentials rotated this week — one reboot each, entirely through the API. No SSH key to any of them exists, because there is nothing for it to open.