The model is an artefact, not a service

Ask most teams where their AI lives and they name a vendor, not a path. The model is a service: a URL, a key, an invoice, a status page to refresh when it breaks. You operate it roughly the way you operate the power grid, which is to say you don't.

Open weights broke that quietly. A quantised checkpoint is a file. You pull it, pin it to an exact revision, sit it on a volume, load it into VRAM when a prompt arrives and unload it when the prompts stop. Every step in that list is artefact management, and none of it is new.

So invert the frame. The model is not a subscription. It is a heavy tool in the workshop crib: signed out against a tag when a job needs it, back on the rack the moment the job ends, and subject to the rack's weight limit like everything else with mass.

Sign the tool out against a tag

A pinned reference is the tag on the tool. Without one you have a rack of unlabelled steel and no way to say which version did the work last Tuesday. So the first move is to read the artefact's real identity from the hub's API rather than from a model card, a parameter count or somebody's blog post:

REPO=unsloth/gpt-oss-20b-GGUF
curl -sf "https://huggingface.co/api/models/$REPO" | jq -r .sha
curl -sf "https://huggingface.co/api/models/$REPO/tree/main?recursive=true" \
  | jq -r '.[] | select(.path | endswith("Q8_0.gguf")) | "\(.path) \(.size) \(.lfs.oid)"'
# prints the branch revision, then "gpt-oss-20b-Q8_0.gguf 12109567168 bcd455d4..." - the bytes and the content hash are the whole identity

Every line that comes back is a filename, its exact size in bytes and its content hash, and those bytes are the only honest basis for deciding whether a checkpoint fits a card. I once skipped the step and put a trillion-parameter model, about 325 GB in its smallest quant, on a shortlist for a 16 GB GPU. Parameter counts lie about bytes. The tree API does not.

Record the whole row, not just the repository. In the lab it is one line in a models.lock that sits next to the manifests:

unsloth/gpt-oss-20b-GGUF | d449b42d93e1c2c7bda5312f5c25c8fb91dfa9b4 | gpt-oss-20b-Q8_0.gguf | 12109567168 | bcd455d4034ec02f71a875b46cb17df44a97911d7258291973be4d21f98329f3

A pinned revision with no hash still trusts whatever the host serves you today. The row is also why the weights are not in the backup set: what gets versioned is the config and the exact reference the weights came from, and losing the volume means re-pulling the same bytes and checking them against the same hash. Reproducible rather than restorable, which is how you already treat a container image.

Budget the memory, because the compute was never scarce

The cost nobody mentions with a hosted model is the first one you meet running your own. The card has 16 GB, and a resident model spends that whether anyone is prompting or not. Weights plus KV cache want to stay under 13.5 GiB so a video encode can still share the silicon; the practical ceiling after driver and graph buffers is 15.0 to 15.3 GiB. Quantisation helps less than the ladder implies. A model whose expert tensors ship natively in MXFP4 spans barely two gigabytes, 10.7 to 12.9 GiB, across its entire GGUF range. Dropping to a low quant buys under half a gigabyte and costs real quality on the dense tensors.

Which makes the return-to-rack automatic rather than clever:

env:
  - name: OLLAMA_KEEP_ALIVE
    value: "15m"
# fifteen idle minutes after the last prompt the model unloads: 4842 MiB resident with a 7b model loaded, 2 MiB after

Note what it does not buy. The card draws 10 W either way, so unloading frees VRAM, not watts. A tool back on the rack still weighs something; it just stops taking up the bench.

Every runtime has the same knob under a different name. The llama.cpp tier in the lab sits behind a swapper with ttl: 900 per model, and llama-server itself now has a router mode that loads models from a directory on request and sleeps them after an idle period (--models-dir, --models-max, --sleep-idle-seconds), so the pattern no longer needs a separate proxy at all. Fifteen minutes is the deliberate compromise: long enough to keep a working session warm, short enough that the card comes back before you forget you borrowed it.

Rearrange the crib without telling anyone

Everything fronts behind one OpenAI-compatible gateway, and that single window is what makes the rest cheap. Pick the runtime per job rather than per allegiance: a pull-by-name server for the dev loop and ordinary serving, a batching engine such as vLLM when many agents hit one card at once, a CPU engine for the machines with no GPU at all. Different tools, one window.

The dividend arrives the day the card is wanted for something else. Scale the backend down and the gateway routes around it:

kubectl -n llm-serving scale deploy/ollama --replicas=0
# the router settings drop a backend after two failed calls and hold it out for 30s; a second backend under the same model name means no caller notices

The GPU is handed back whole, and no application upstream is told, because no application upstream was ever told a backend existed. They only knew the gateway.

Here is the one that cost an evening. That routing has a price when the second backend is off rather than failing: an entry for a powered-down node left in the list is shuffled into every request, burns a connect timeout and two retries, drops out for its cooldown, then re-enters rotation and does it all again. Nothing errors. The assistant is just slow, and the logs say nothing about why. A backend that is off comes out of the list; the config is a statement of what exists, not of what might.

There is a matching discipline on the way in. The serving namespace has DNS-only egress and cannot reach a model hub at all, and punching a hole for one hostname is the wrong answer. Stage the file from the operator box, copy it in over a read-only mount, verify the hash in-cluster before anything can load it. The server never dials the hub; it mounts what the lockfile already named.

The principle

Strip the mystique and an open-weights model is a large, well-labelled file that is expensive to keep loaded. It obeys artefact rules because it is one, and every rule it needs was written years ago for container images: an exact reference, a versioned config that names it, a budget for what it occupies while it runs.

Nothing here needed new rules. It needed the old ones applied without awe.

Live on a 16 GB RTX 5080 node: pinned checkpoints on a volume, one lockfile row per file, a LiteLLM gateway in front, and a fifteen-minute keep-alive that puts the tool back on the rack.