#Tag
Posts tagged "homelab"
413 restarts, and not one could reach the fault
A liveness probe restarted a container over and over without once repairing it, because the broken state lived in the pod's network namespace rather than inside the container. Detection and remediation are separate jobs, and a restart only heals when the fault sits inside the thing being restarted.
One way in, and a dead route looks like a dead estate
A broken route into a network is indistinguishable from every service in it being down, because the evidence available from outside is identical in both cases. The fix is an order of diagnosis - prove the service from a chair that is not yours, then walk the path - and a gate that is never a tenant of the building it lets you into.
A mirror saves the download, not the dependency
A pull-through mirror turns repeated image downloads into cache hits, but a cache that re-checks tags and falls back to the origin has not removed the upstream from your availability story. What it must do, and does not by default, is serve the exact manifest it was given, because an image digest is the hash of that manifest.
A typography rule turned out to be a supply-chain control
Zero-width Unicode lets an instruction file carry text a reviewer cannot see and a model's tokeniser still receives, and a campaign reported in May 2026 planted files built that way. The ASCII ban I wrote for tidy prose catches it, provided I remember how little a clean result proves.
Apps should dial the switchboard, never the model
Wiring each application straight at a model server's URL works on the first try, and from then on every change to the model layer is a change to every consumer. One OpenAI-compatible gateway in front turns backends into lines that can be added, drained or replaced without editing the consumers that dial it, provided you are honest about what the gateway does and does not carry across.
A healthy tunnel can still break cluster DNS
Twice a shared pod in my lab has failed on a name it could not resolve, and twice the component raising the alarm was working correctly. Both times the resolver every container in that pod read had been rewritten by the tunnel container beside them, and the fastest route to the cause was testing the dependency instead of the component reporting the fault.
A finished scan can have walked nothing
A library scan answered 200 and its scanner logged a launch and a finish about eighty milliseconds apart with zero files found, because one parameter was an empty array where a folder path belonged. The tool had done exactly what it was asked. Only an inventory taken outside it, set against its own records, showed two libraries 164 files short.
The server took the new limit. The node kept the old one.
I raised a transcode node's worker limit through the server's database endpoint, the server reported the new value, and the node kept behaving as before until I restarted it. That gave me a recovery step, not the reason the update never reached the running node.
A healthy deployment is not a reachable one
On 2026-09-02 a new service behind my lab's Traefik gateway was Ready by every check it could run on itself, and every request from outside died at the gateway's own egress policy. Readiness, routing and network policy each cover a different part of the path a request takes, and only the whole path is reachability.
My flow named its encoder, and the Intel node paid for it
A shared Tdarr flow that stored NVENC as its encoder failed on my Intel node with an NVIDIA runtime error. Setting the encoder to auto let each node's probe choose, which fixed my case. Pinning is still a valid choice when you want one specific encoder and every node has it.
The model is an artefact, not a service
An open-weights model is a file you pull, pin and put on a volume, so it obeys the artefact rules you already run rather than a new discipline called AI operations. The only new budget is VRAM, and the fix for that is unloading the tool the moment nobody is asking it questions.
Two clusters, one view, and the label that decides whether it works
Forwarding metrics from every cluster to one Prometheus does not give you a fleet view; it gives you one Prometheus full of series that cannot say where they came from. The whole design turns on a single label, and the part everyone misses is that the hub's own scrapes never pass through the forwarding config that stamps it.
NVENC and CUDA are different silicon - so share the card
An LLM and a video transcoder both want a GPU, and the obvious answer is 'buy two.' But a modern GPU isn't one resource - the video encoder is a separate chip from the compute cores. Here's how one RTX 5080 runs local inference and AV1 transcoding at the same time.
A new model is a new hire, not a new file
A model pulled from the hub is not just weights - it ships tokeniser code, an executable chat template, loaders gated by trust_remote_code, sometimes its own runtime, and the ecosystem's default is to run all of it beside your credentials. So every new model starts on probation: a hardware-isolated microVM with its own guest kernel, a default-deny network, zero credentials, and promotion to the standard serving tier only when the observed record earns it.
Six days of nothing, and nothing noticed
A VPN tunnel died and downloads stopped for six days. Every container stayed running, every check passed, and the platform was entirely satisfied, because nothing was watching the thing that had actually failed.
The workbench gets a clone of the brain, never the keys
A browser IDE you can reach from the couch or a phone is a legitimate want, and the obvious build, hosting it where the keys already live, quietly parks the whole estate behind a browser tab. Price the stolen session first, then hand the new surface copies of everything and originals of nothing, so the worst case is a git revert instead of a lost fleet.
The best migrations delete more than they install
Nobody chose our network stack - it accreted one default at a time, and every component was one more thing to version, patch, debug and alert on. Landing on Cilium wasn't an adoption; it was a demolition schedule, and the stack left standing is smaller than the one we started with.
SSO is for people, not for machines
When an app grows a human face on an interface machines also depend on, one auth layer over everything either breaks the machines or quietly weakens the human door to let them through. The pattern that holds: a single identity-provider proxy in front of every human door, while machine paths keep their own scoped credentials - route by audience, not by app.
Powered off is a state, not an incident
Three of this fleet's six machines were upgraded, verified healthy, and then deliberately shut down the same night, and the books record a state, not a failure. When a node's entire existence is a rebuildable document, powered off becomes something you schedule, not something you fear.
Stage every lock before you turn one
Turning off anonymous pulls on the registry a fleet boots from is a change that only fails at the next boot, and a node that can't pull at boot may not come back to tell you. So the flip is split in time: the credential staged inert in every node's config, each node rebooted and proven with a real pull, and the door closed only when the last key has turned.
My servers don't have SSH, and that's the feature
Every box in this fleet runs an OS with no shell, no package manager, and no SSH daemon - the entire machine is an API with a declarative config. It sounds like giving up control. It's the opposite: you can't drift what you can't touch.
The bastion is a pattern, not a box
Admin credentials accumulate on whatever machine you happen to work from, until your daily driver is the real control plane of the fleet. A bastion fixes that, not by being special hardware, but by being the one place the keys live and the discipline that they never leave it.
An agent should never hold the key it's using
You want an AI agent that can actually do things - call APIs, touch real data. You also don't fully trust it. The resolution isn't a better sandbox; it's making sure the agent never possesses a credential at all. A broker holds the keys, mints short-lived capabilities, and gates every write behind a human. Here's the pattern.
A backup you haven't restored is a rumour
Backup jobs report success every night for years, and none of those green ticks proves the one thing backups exist for. The only receipt is a restore drill: files pulled from the repository, checked, and running. Here's the discipline, and the failure it caught.
Shipping this site: GitOps from a homelab to the public internet
How this portfolio is built and served - Astro to a container image, a self-hosted Gitea registry, Argo CD, and a Cloudflare Tunnel - with security as acceptance criteria, not polish.