#Tag
Posts tagged "kubernetes"
413 restarts, and not one could reach the fault
A liveness probe restarted a container over and over without once repairing it, because the broken state lived in the pod's network namespace rather than inside the container. Detection and remediation are separate jobs, and a restart only heals when the fault sits inside the thing being restarted.
Liveness probes kill things that would have lived
An edge blip set my tunnel client's health endpoint failing, and both replicas were restarted once it outlasted the budget a liveness probe on its defaults allowed, while the process was reconnecting on its own. A liveness probe should catch a deadlock, not interrupt a recovery, so its budget has to outlast the recovery it watches.
A health check cannot prove a reboot happened
Automation that confirms a restart by polling for a healthy API is asking a question the old boot answers just as cheerfully as the new one. Gate a transition on evidence only the new state can produce: for a reboot, a boot ID that differs from the one you recorded before the change.
Apps should dial the switchboard, never the model
Wiring each application straight at a model server's URL works on the first try, and from then on every change to the model layer is a change to every consumer. One OpenAI-compatible gateway in front turns backends into lines that can be added, drained or replaced without editing the consumers that dial it, provided you are honest about what the gateway does and does not carry across.
fsGroup is a request, not a guarantee
A non-root pod crashlooped on permission denied with a textbook securityContext, because the volume plugin behind a host-directory-backed claim carries no ownership pass and nothing says so. The field validated, the ownership change never happened, and only the directory reached through the application's own mount could settle it.
The model is an artefact, not a service
An open-weights model is a file you pull, pin and put on a volume, so it obeys the artefact rules you already run rather than a new discipline called AI operations. The only new budget is VRAM, and the fix for that is unloading the tool the moment nobody is asking it questions.
If your automation opens an SSH session, it isn't automation
A play that opens a shell and greps the output is screen-scraping with a schedule attached, and it reports changed on every run because it has no idea whether it changed anything. The same work rebuilt against a typed API buys idempotency, a real preview and a change count you can believe.
Two clusters, one view, and the label that decides whether it works
Forwarding metrics from every cluster to one Prometheus does not give you a fleet view; it gives you one Prometheus full of series that cannot say where they came from. The whole design turns on a single label, and the part everyone misses is that the hub's own scrapes never pass through the forwarding config that stamps it.
NVENC and CUDA are different silicon - so share the card
An LLM and a video transcoder both want a GPU, and the obvious answer is 'buy two.' But a modern GPU isn't one resource - the video encoder is a separate chip from the compute cores. Here's how one RTX 5080 runs local inference and AV1 transcoding at the same time.
Untrusted code belongs in a VM, not a namespace
A container isn't a security boundary - it's a process sharing the host kernel, wearing namespaces. For code you genuinely don't trust, that's not enough. Kata Containers give each pod its own microVM and guest kernel, with the kubectl ergonomics intact. Here's why, and what it costs.
The workbench gets a clone of the brain, never the keys
A browser IDE you can reach from the couch or a phone is a legitimate want, and the obvious build, hosting it where the keys already live, quietly parks the whole estate behind a browser tab. Price the stolen session first, then hand the new surface copies of everything and originals of nothing, so the worst case is a git revert instead of a lost fleet.
The best migrations delete more than they install
Nobody chose our network stack - it accreted one default at a time, and every component was one more thing to version, patch, debug and alert on. Landing on Cilium wasn't an adoption; it was a demolition schedule, and the stack left standing is smaller than the one we started with.
SSO is for people, not for machines
When an app grows a human face on an interface machines also depend on, one auth layer over everything either breaks the machines or quietly weakens the human door to let them through. The pattern that holds: a single identity-provider proxy in front of every human door, while machine paths keep their own scoped credentials - route by audience, not by app.
Stage every lock before you turn one
Turning off anonymous pulls on the registry a fleet boots from is a change that only fails at the next boot, and a node that can't pull at boot may not come back to tell you. So the flip is split in time: the credential staged inert in every node's config, each node rebooted and proven with a real pull, and the door closed only when the last key has turned.
My servers don't have SSH, and that's the feature
Every box in this fleet runs an OS with no shell, no package manager, and no SSH daemon - the entire machine is an API with a declarative config. It sounds like giving up control. It's the opposite: you can't drift what you can't touch.
Observe first, deny second
Everyone writes network policy from the architecture diagram, and the diagram is always wrong. The only allowlist that survives contact with production is one written from the flows you actually watched - applied out-of-band, proven enforcing, and only then handed to GitOps.
SNAT ate my source IP
A LoadBalancer service with the default traffic policy rewrites every incoming packet's source to the node's own address, so by the time a network policy sees it, the real client is gone. You cannot allowlist a sender the network has already erased.
A backup you haven't restored is a rumour
Backup jobs report success every night for years, and none of those green ticks proves the one thing backups exist for. The only receipt is a restore drill: files pulled from the repository, checked, and running. Here's the discipline, and the failure it caught.
The first secret is the one you can't commit
Secrets management has a bootstrap paradox: the credential that pulls every other secret can't itself live in git. Here's how a homelab fleet breaks the cycle - zero plaintext secrets in any repo, and a clean rule for which is the one exception.
SSO is a perimeter decision, not a login box
Putting single sign-on in front of a fleet of self-hosted apps isn't about a nicer login screen - it's about where your trust boundary lives, and how it fails.
The most secure inbound port is the one you never open
Exposing self-hosted services to the internet with zero open ports, zero port-forwarding, and the origin's IP never leaving the building.
Every pod holds a key to a door it never opens
Least privilege for Kubernetes workloads doesn't start with an RBAC role - it starts with revoking the API token every pod silently carries, then layering identity, non-root, and Pod Security on top.
Init-gating GPU readiness on Kubernetes
The single highest-leverage reliability fix for edge GPU workloads: never let an inference pod schedule before the GPU is actually ready.