*~/jwright/blog
Notes from the build.
Lessons from edge Kubernetes, GPUs, and running infrastructure like it matters.
The first secret is the one you can't commit
Secrets management has a bootstrap paradox: the credential that pulls every other secret can't itself live in git. Here's how a homelab fleet breaks the cycle - zero plaintext secrets in any repo, and a clean rule for which is the one exception.
Read the post →
Untrusted code belongs in a VM, not a namespace
A container isn't a security boundary - it's a process sharing the host kernel, wearing namespaces. For code you genuinely don't trust, that's not enough. Kata Containers give each pod its own microVM and guest kernel, with the kubectl ergonomics intact. Here's why, and what it costs.
Consistency by construction beats consistency by discipline
One address had to appear in the application config, the network attachment and two policy objects - four artefacts, four update paths. Rendering every one of them from a single required value made divergence something the tooling can no longer express.
A new model is a new hire, not a new file
A model pulled from the hub is not just weights - it ships tokeniser code, an executable chat template, loaders gated by trust_remote_code, sometimes its own runtime, and the ecosystem's default is to run all of it beside your credentials. So every new model starts on probation: a hardware-isolated microVM with its own guest kernel, a default-deny network, zero credentials, and promotion to the standard serving tier only when the observed record earns it.
Six days of nothing, and nothing noticed
A VPN tunnel died and downloads stopped for six days. Every container stayed running, every check passed, and the platform was entirely satisfied, because nothing was watching the thing that had actually failed.
The workbench gets a clone of the brain, never the keys
A browser IDE you can reach from the couch or a phone is a legitimate want, and the obvious build, hosting it where the keys already live, quietly parks the whole estate behind a browser tab. Price the stolen session first, then hand the new surface copies of everything and originals of nothing, so the worst case is a git revert instead of a lost fleet.
The best migrations delete more than they install
Nobody chose our network stack - it accreted one default at a time, and every component was one more thing to version, patch, debug and alert on. Landing on Cilium wasn't an adoption; it was a demolition schedule, and the stack left standing is smaller than the one we started with.
SSO is for people, not for machines
When an app grows a human face on an interface machines also depend on, one auth layer over everything either breaks the machines or quietly weakens the human door to let them through. The pattern that holds: a single identity-provider proxy in front of every human door, while machine paths keep their own scoped credentials - route by audience, not by app.
Powered off is a state, not an incident
Three of this fleet's six machines were upgraded, verified healthy, and then deliberately shut down the same night, and the books record a state, not a failure. When a node's entire existence is a rebuildable document, powered off becomes something you schedule, not something you fear.
Stage every lock before you turn one
Turning off anonymous pulls on the registry a fleet boots from is a change that only fails at the next boot, and a node that can't pull at boot may not come back to tell you. So the flip is split in time: the credential staged inert in every node's config, each node rebooted and proven with a real pull, and the door closed only when the last key has turned.
My servers don't have SSH, and that's the feature
Every box in this fleet runs an OS with no shell, no package manager, and no SSH daemon - the entire machine is an API with a declarative config. It sounds like giving up control. It's the opposite: you can't drift what you can't touch.
Observe first, deny second
Everyone writes network policy from the architecture diagram, and the diagram is always wrong. The only allowlist that survives contact with production is one written from the flows you actually watched - applied out-of-band, proven enforcing, and only then handed to GitOps.
SNAT ate my source IP
A LoadBalancer service with the default traffic policy rewrites every incoming packet's source to the node's own address, so by the time a network policy sees it, the real client is gone. You cannot allowlist a sender the network has already erased.
The bastion is a pattern, not a box
Admin credentials accumulate on whatever machine you happen to work from, until your daily driver is the real control plane of the fleet. A bastion fixes that, not by being special hardware, but by being the one place the keys live and the discipline that they never leave it.
An agent should never hold the key it's using
You want an AI agent that can actually do things - call APIs, touch real data. You also don't fully trust it. The resolution isn't a better sandbox; it's making sure the agent never possesses a credential at all. A broker holds the keys, mints short-lived capabilities, and gates every write behind a human. Here's the pattern.
The cert that couldn't see its own proof
Let's Encrypt DNS-01 posted its proof to the public internet, and then checked for it through the LAN's split-horizon DNS, which will never carry it. The fix is one line: make the self-check look where the CA looks, not where you live.
A backup you haven't restored is a rumour
Backup jobs report success every night for years, and none of those green ticks proves the one thing backups exist for. The only receipt is a restore drill: files pulled from the repository, checked, and running. Here's the discipline, and the failure it caught.
SSO is a perimeter decision, not a login box
Putting single sign-on in front of a fleet of self-hosted apps isn't about a nicer login screen - it's about where your trust boundary lives, and how it fails.
The most secure inbound port is the one you never open
Exposing self-hosted services to the internet with zero open ports, zero port-forwarding, and the origin's IP never leaving the building.
Every pod holds a key to a door it never opens
Least privilege for Kubernetes workloads doesn't start with an RBAC role - it starts with revoking the API token every pod silently carries, then layering identity, non-root, and Pod Security on top.
Shipping this site: GitOps from a homelab to the public internet
How this portfolio is built and served - Astro to a container image, a self-hosted Gitea registry, Argo CD, and a Cloudflare Tunnel - with security as acceptance criteria, not polish.
Init-gating GPU readiness on Kubernetes
The single highest-leverage reliability fix for edge GPU workloads: never let an inference pod schedule before the GPU is actually ready.