~/jwright
← All posts
SECURITY SERIES · 06··6 min read

The workbench gets a clone of the brain, never the keys

A luminous reading room of copied pages, thin circuit traces running back to a sealed vault where the original documents rest untouched

You want your development environment everywhere. VS Code in a browser tab, reachable from the couch on the LAN and from a phone across the mesh VPN, with git and the daily tooling already inside. It's a legitimate want - it's how odd half-hours turn into finished work.

And the first instinct is obvious: host it on the operator box. That machine already has everything - every repo cloned, every context wired, every login warm. One container, one hostname, done by the weekend.

Read that instinct back slowly, because of what it actually proposes. The operator box is the one machine that can drive the entire fleet: the master decryption key for every secret, the admin kubeconfigs, the SSH keys. "Host it there" means parking all of that behind a browser tab - one stolen session away from anyone who ever finds the door. I've argued before that the fleet's keys should exist in exactly one place; this was the first time a convenience asked to move in with them. The design that survived the red-team gives the convenience something else entirely: a clone of the operator's brain - the repos, the tooling, the logins, furnished fresh. Never the keys.

One guarded door to a pod that holds copies and its own scoped identity; the operator's keys sit outside the frame entirely

Price a stolen session before you build the surface

The question that shaped this build wasn't "how do we secure it?" It was blunter: assume the worst has already happened - the session is fully stolen, an attacker sitting in the IDE as you, and write down what they now hold.

Against the first sketch, the one that ran on the operator host, the red-team returned three separate critical findings, all of them consequences of proximity to that host's keys. Each would have needed its own mitigation, its own monitoring, its own upkeep. Then the design moved to a disposable pod that inherits nothing from the operator host, and all three findings died in the same moment. Not mitigated - deleted. There was nothing left to steal along those routes, because nothing had been put there.

That is the cheapest security work I have ever done: three critical findings closed by a relocation, before a single line of configuration existed. Blast radius is a design input. Treat it as an audit output and you'll be retrofitting forever.

Clone everything, inherit nothing

Think of the workbench as a reading room attached to an archive. A reading room is genuinely useful - every document laid out on the desks, annotated, workable, open late. But the pages on the desks are photocopies. The originals stay in the vault, and no reader's card opens it.

The pod starts empty. On first run I furnished it the way you'd stock a reading room: cloned the repos into it, generated a fresh git key inside it - registered in its own name, revocable on its own, and signed in to the daily tooling so those logins persist on the pod's own volume. Nothing is mounted from the operator host. Nothing is inherited. And the one thing deliberately kept out is the master secrets key - the key that turns the encrypted blobs in those repos into live fleet credentials. The workbench holds a copy of every document; the pages that matter are ciphertext to it.

One call in there deserves honesty, because it was a real trade. That git identity has account-level access: the workbench can push, including to the repo the fleet reconciles from. That is genuine power in a convenience surface, and I granted it with eyes open - partly because the door in front of it is layered, and mostly because of what the worst case is. A hostile push is loud. It lands in git history, attributed and timestamped, and git revert undoes it. Set that against the alternative worst case - silent possession of every credential in the estate, and "attacker can push to some repos" is a failure you recover from, audit, and learn from. The other one is a failure you rebuild from.

Put everything behind one proven door

The front of the workbench is SSO-only, through the identity provider's proxy, and everything means everything. The route sends 100% of paths through the proxy, with no carve-outs, because the obvious carve-out is a hole: the terminal speaks over a WebSocket, and a WebSocket path excused from authentication is an unauthenticated shell. So it was proven, not assumed - probe the root unauthenticated, probe the terminal's WebSocket, probe the static assets and the health endpoint, and every one of them 302s to the login page.

Behind the SSO sits the application's own password, injected from the secrets manager rather than living in any repo. Belt behind braces: a bug in the proxy still lands on a locked application.

Under both sits a default-deny network policy. The pod can reach the git server, DNS, and its API endpoints over 443 - nothing else. That floor is real enough to bite: the git server's own policy had to grant the workbench an ingress seat by name, and the very first clone hung until it did. The hang was the policy working.

Just as important are the doors that don't exist. A direct LAN bind and a separate mesh-VPN serving path were both sketched, and both cut - the one guarded door already serves the couch and the phone alike. Every additional entrance is an additional thing to prove, forever.

Work backwards from a total compromise

Every new convenience surface - a browser IDE, a remote dashboard, a sync agent - should be designed backwards from one number: what a total compromise of it yields. Compute that before the surface exists, while the answer is still a design choice. Then engineer it down: clones instead of originals, an identity scoped to the surface and revocable without ceremony, doors layered so no single failure is enough. Only when the worst case is something you could live through do you get to enjoy the convenience, and then you get to enjoy it completely, from any couch you like.

A reading room earns its comfort by holding photocopies. You can deface every page on every desk and the archive survives, because the originals were never in the room.

Live at code.bztmon.org: a code-server pod on one Talos node, its home a 20Gi volume of cloned repos, its own git key and its own logins - behind an Authentik proxy that 302s every path, WebSockets included, to login, above a default-deny network policy. The fleet's age key has never entered it.