*~/jwright/blog
Notes from the build.
Lessons from edge Kubernetes, GPUs, and running infrastructure like it matters.
The first secret is the one you can't commit
Secrets management has a bootstrap paradox: the credential that pulls every other secret can't itself live in git. Here's how a homelab fleet breaks the cycle - zero plaintext secrets in any repo, and a clean rule for which is the one exception.
Read the post →
413 restarts, and not one could reach the fault
A liveness probe restarted a container over and over without once repairing it, because the broken state lived in the pod's network namespace rather than inside the container. Detection and remediation are separate jobs, and a restart only heals when the fault sits inside the thing being restarted.
One way in, and a dead route looks like a dead estate
A broken route into a network is indistinguishable from every service in it being down, because the evidence available from outside is identical in both cases. The fix is an order of diagnosis - prove the service from a chair that is not yours, then walk the path - and a gate that is never a tenant of the building it lets you into.
A mirror saves the download, not the dependency
A pull-through mirror turns repeated image downloads into cache hits, but a cache that re-checks tags and falls back to the origin has not removed the upstream from your availability story. What it must do, and does not by default, is serve the exact manifest it was given, because an image digest is the hash of that manifest.
Three empty builds before I read the kernel's file
A per-engine GPU exporter for two consumer Intel Arc cards took five builds in one evening, and the three before the first real row came back empty because each trusted a reading borrowed from somewhere else. The row arrived only after the kernel's own file was read off the working card, which is where any parser for an interface you have not read yourself should start.
Liveness probes kill things that would have lived
An edge blip set my tunnel client's health endpoint failing, and both replicas were restarted once it outlasted the budget a liveness probe on its defaults allowed, while the process was reconnecting on its own. A liveness probe should catch a deadlock, not interrupt a recovery, so its budget has to outlast the recovery it watches.
The form supplied defaults my script never sent
Create an OAuth2 provider through Authentik's admin form and logins work; create it in the admin shell and every login comes back as a malformed request, because the form was ticking boxes the model leaves empty. Automation that writes underneath a form inherits the form's job, and a working object shows what that job includes without deciding it for the next app.
A typography rule turned out to be a supply-chain control
Zero-width Unicode lets an instruction file carry text a reviewer cannot see and a model's tokeniser still receives, and a campaign reported in May 2026 planted files built that way. The ASCII ban I wrote for tidy prose catches it, provided I remember how little a clean result proves.
A health check cannot prove a reboot happened
Automation that confirms a restart by polling for a healthy API is asking a question the old boot answers just as cheerfully as the new one. Gate a transition on evidence only the new state can produce: for a reboot, a boot ID that differs from the one you recorded before the change.
Apps should dial the switchboard, never the model
Wiring each application straight at a model server's URL works on the first try, and from then on every change to the model layer is a change to every consumer. One OpenAI-compatible gateway in front turns backends into lines that can be added, drained or replaced without editing the consumers that dial it, provided you are honest about what the gateway does and does not carry across.
A healthy tunnel can still break cluster DNS
Twice a shared pod in my lab has failed on a name it could not resolve, and twice the component raising the alarm was working correctly. Both times the resolver every container in that pod read had been rewritten by the tunnel container beside them, and the fastest route to the cause was testing the dependency instead of the component reporting the fault.
fsGroup is a request, not a guarantee
A non-root pod crashlooped on permission denied with a textbook securityContext, because the volume plugin behind a host-directory-backed claim carries no ownership pass and nothing says so. The field validated, the ownership change never happened, and only the directory reached through the application's own mount could settle it.
A finished scan can have walked nothing
A library scan answered 200 and its scanner logged a launch and a finish about eighty milliseconds apart with zero files found, because one parameter was an empty array where a folder path belonged. The tool had done exactly what it was asked. Only an inventory taken outside it, set against its own records, showed two libraries 164 files short.
The server took the new limit. The node kept the old one.
I raised a transcode node's worker limit through the server's database endpoint, the server reported the new value, and the node kept behaving as before until I restarted it. That gave me a recovery step, not the reason the update never reached the running node.
A healthy deployment is not a reachable one
On 2026-09-02 a new service behind my lab's Traefik gateway was Ready by every check it could run on itself, and every request from outside died at the gateway's own egress policy. Readiness, routing and network policy each cover a different part of the path a request takes, and only the whole path is reachability.
A metric can be accurate and still be history
After a maintenance window my not-ready panel went red over a cluster where nothing was wrong. It was counting Succeeded pods still held in the API. The fix is an exclusion filtered by value for readiness and a separate freshness check for backups, and both expressions now have unit tests.
My flow named its encoder, and the Intel node paid for it
A shared Tdarr flow that stored NVENC as its encoder failed on my Intel node with an NVIDIA runtime error. Setting the encoder to auto let each node's probe choose, which fixed my case. Pinning is still a valid choice when you want one specific encoder and every node has it.
The model is an artefact, not a service
An open-weights model is a file you pull, pin and put on a volume, so it obeys the artefact rules you already run rather than a new discipline called AI operations. The only new budget is VRAM, and the fix for that is unloading the tool the moment nobody is asking it questions.
If your automation opens an SSH session, it isn't automation
A play that opens a shell and greps the output is screen-scraping with a schedule attached, and it reports changed on every run because it has no idea whether it changed anything. The same work rebuilt against a typed API buys idempotency, a real preview and a change count you can believe.
Two clusters, one view, and the label that decides whether it works
Forwarding metrics from every cluster to one Prometheus does not give you a fleet view; it gives you one Prometheus full of series that cannot say where they came from. The whole design turns on a single label, and the part everyone misses is that the hub's own scrapes never pass through the forwarding config that stamps it.
NVENC and CUDA are different silicon - so share the card
An LLM and a video transcoder both want a GPU, and the obvious answer is 'buy two.' But a modern GPU isn't one resource - the video encoder is a separate chip from the compute cores. Here's how one RTX 5080 runs local inference and AV1 transcoding at the same time.
The alert that could never fire
A readiness alert that passed review was watching for a series the remote cluster's collector filters out before it ever leaves. An alert you never test is a belief, not a control.
Every build succeeded. The dashboard said they all failed.
A CI runner executed flawlessly for a month - images pushed, deploy branches landed - while reporting failure to the dashboard every time. A three-post release sat stranded for a fortnight because the status was believed over the artefacts.
Untrusted code belongs in a VM, not a namespace
A container isn't a security boundary - it's a process sharing the host kernel, wearing namespaces. For code you genuinely don't trust, that's not enough. Kata Containers give each pod its own microVM and guest kernel, with the kubectl ergonomics intact. Here's why, and what it costs.
Consistency by construction beats consistency by discipline
One address had to appear in the application config, the network attachment and two policy objects - four artefacts, four update paths. Rendering every one of them from a single required value made divergence something the tooling can no longer express.
A new model is a new hire, not a new file
A model pulled from the hub is not just weights - it ships tokeniser code, an executable chat template, loaders gated by trust_remote_code, sometimes its own runtime, and the ecosystem's default is to run all of it beside your credentials. So every new model starts on probation: a hardware-isolated microVM with its own guest kernel, a default-deny network, zero credentials, and promotion to the standard serving tier only when the observed record earns it.
Six days of nothing, and nothing noticed
A VPN tunnel died and downloads stopped for six days. Every container stayed running, every check passed, and the platform was entirely satisfied, because nothing was watching the thing that had actually failed.
The workbench gets a clone of the brain, never the keys
A browser IDE you can reach from the couch or a phone is a legitimate want, and the obvious build, hosting it where the keys already live, quietly parks the whole estate behind a browser tab. Price the stolen session first, then hand the new surface copies of everything and originals of nothing, so the worst case is a git revert instead of a lost fleet.
The best migrations delete more than they install
Nobody chose our network stack - it accreted one default at a time, and every component was one more thing to version, patch, debug and alert on. Landing on Cilium wasn't an adoption; it was a demolition schedule, and the stack left standing is smaller than the one we started with.
SSO is for people, not for machines
When an app grows a human face on an interface machines also depend on, one auth layer over everything either breaks the machines or quietly weakens the human door to let them through. The pattern that holds: a single identity-provider proxy in front of every human door, while machine paths keep their own scoped credentials - route by audience, not by app.
Powered off is a state, not an incident
Three of this fleet's six machines were upgraded, verified healthy, and then deliberately shut down the same night, and the books record a state, not a failure. When a node's entire existence is a rebuildable document, powered off becomes something you schedule, not something you fear.
Stage every lock before you turn one
Turning off anonymous pulls on the registry a fleet boots from is a change that only fails at the next boot, and a node that can't pull at boot may not come back to tell you. So the flip is split in time: the credential staged inert in every node's config, each node rebooted and proven with a real pull, and the door closed only when the last key has turned.
My servers don't have SSH, and that's the feature
Every box in this fleet runs an OS with no shell, no package manager, and no SSH daemon - the entire machine is an API with a declarative config. It sounds like giving up control. It's the opposite: you can't drift what you can't touch.
Observe first, deny second
Everyone writes network policy from the architecture diagram, and the diagram is always wrong. The only allowlist that survives contact with production is one written from the flows you actually watched - applied out-of-band, proven enforcing, and only then handed to GitOps.
SNAT ate my source IP
A LoadBalancer service with the default traffic policy rewrites every incoming packet's source to the node's own address, so by the time a network policy sees it, the real client is gone. You cannot allowlist a sender the network has already erased.
The bastion is a pattern, not a box
Admin credentials accumulate on whatever machine you happen to work from, until your daily driver is the real control plane of the fleet. A bastion fixes that, not by being special hardware, but by being the one place the keys live and the discipline that they never leave it.
An agent should never hold the key it's using
You want an AI agent that can actually do things - call APIs, touch real data. You also don't fully trust it. The resolution isn't a better sandbox; it's making sure the agent never possesses a credential at all. A broker holds the keys, mints short-lived capabilities, and gates every write behind a human. Here's the pattern.
The cert that couldn't see its own proof
Let's Encrypt DNS-01 posted its proof to the public internet, and then checked for it through the LAN's split-horizon DNS, which will never carry it. The fix is one line: make the self-check look where the CA looks, not where you live.
A backup you haven't restored is a rumour
Backup jobs report success every night for years, and none of those green ticks proves the one thing backups exist for. The only receipt is a restore drill: files pulled from the repository, checked, and running. Here's the discipline, and the failure it caught.
SSO is a perimeter decision, not a login box
Putting single sign-on in front of a fleet of self-hosted apps isn't about a nicer login screen - it's about where your trust boundary lives, and how it fails.
The most secure inbound port is the one you never open
Exposing self-hosted services to the internet with zero open ports, zero port-forwarding, and the origin's IP never leaving the building.
Every pod holds a key to a door it never opens
Least privilege for Kubernetes workloads doesn't start with an RBAC role - it starts with revoking the API token every pod silently carries, then layering identity, non-root, and Pod Security on top.
Shipping this site: GitOps from a homelab to the public internet
How this portfolio is built and served - Astro to a container image, a self-hosted Gitea registry, Argo CD, and a Cloudflare Tunnel - with security as acceptance criteria, not polish.
Init-gating GPU readiness on Kubernetes
The single highest-leverage reliability fix for edge GPU workloads: never let an inference pod schedule before the GPU is actually ready.