A new model is a new hire, not a new file

Pull a model from the hub and notice what your own head does with it. A directory of large binary files arrives, so it gets filed under data. Weights. Tensors. Numbers, and numbers can't run.

Except that isn't what you downloaded. A model artefact ships tokeniser code. It ships a chat template - an executable template the runtime evaluates on every request. It often ships a loader gated by a flag whose name confesses everything, trust_remote_code=True, and sometimes it effectively ships its own runtime, pulled as a container image from someone else's registry. The ecosystem's default is to run all of this with cluster-level convenience: the same kernel as everything else you host, the same namespace, a service-account token mounted at the usual path. Treat a fresh model as data and you will, sooner or later, execute a stranger's Python next to your credentials.

I've argued before that untrusted code belongs in a VM, not a namespace; the downloaded model is that argument's most routinely ignored case. So the operating rule in this fleet is a hiring rule, not a file-handling one: new models start on probation. You didn't add an asset to storage. You hired a stranger off the internet, and a new hire doesn't get keys to anything on day one.

Admit what you actually downloaded

Consider what you'd normally know about a new starter: references you can call, a work history someone vouches for, an interview where you watched them think. Now inventory what you know about a model pulled from a public hub: an account name, a download count, a README written by the account. That's the whole CV, and none of it is checkable.

Meanwhile the artefact's executable surface is wider than most people's mental picture of it. The older checkpoint formats are pickles, and unpickling is code execution - safetensors exists precisely because "loading the weights" used to mean "running a program". Custom architectures ask for trust_remote_code, which imports and runs whatever Python the repository carries, in your process, with your permissions. Even the innocuous chat template is a small program, executed on every request. Little of this is malice; nearly all of it is engineering convenience. But convenience is how a stranger's code ends up running beside a token that can list every secret in the namespace.

You cannot call this candidate's references. The only reference check available is watching it work, which is what a probation period is.

Give the stranger a supervised shift

The probation tier here is not a policy document; it's a namespace, and everything in it is live and verified. Kata Containers is baked into the node's install image, and a pod that sets runtimeClassName: kata doesn't get a slice of the host kernel - it boots inside its own KVM microVM, with its own guest kernel. Not a claim from the docs: run uname -r inside such a pod and it reads 6.18.28 while the node underneath runs 6.18.34. A workload that escapes its container has escaped into a guest VM, with a hardware boundary still between it and the node.

A Sandbox resource (kubernetes-sigs/agent-sandbox) wraps that pod and owns its lifecycle, and the namespace floor does the supervising. The network is default-deny with zero peers - a fresh sandbox can pull its image and reach nothing else until someone writes down, explicitly, the handful of addresses it's allowed. Admission rejects any pod that tries to start in that namespace without the microVM runtime, so nobody can absent-mindedly schedule a shortcut. And the tier holds no credentials at all, by design: anything a workload legitimately needs is asked for through the broker layer above it, which holds the real keys - the pattern from an agent should never hold the key it's using.

That's a supervised shift in full. Escorted everywhere it goes, nothing in its pockets, every outside contact through a chaperone, and the building stays standing even if the new starter turns out to be hostile.

Promote on the record, not the calendar

Here is the honest line between proven and doctrine. The machinery above is live: the microVM boots, the guest kernel differs, the deny floor holds - all verified. The ladder is the operating rule that machinery was built to serve, and it goes like this.

A candidate model runs its evaluation harness inside the microVM, pinned by digest - the exact artefact and the exact runtime, not a tag that can drift under you. Its declared needs are written down first: these endpoints, this storage, nothing else. Then the record accumulates over real workloads. Egress matches the declaration, or it doesn't. Behaviour matches expectation, or it doesn't - no surprise network, no surprise syscalls, no creative interpretation of its job. When the record is long enough and boring enough, probation ends: the model is made permanent and moves to the standard serving tier behind the gateway, where the fast path lives.

The quiet half of the rule is the better half: anything unproven simply stays in the VM. Forever, if need be. Permanent probation costs almost nothing - the model still works, still answers, still earns its keep on supervised shifts. Promotion is a decision you can decline to make indefinitely, and the candidate has no grounds to complain.

Pay the probation tax without flinching

None of this is free. A microVM boots in seconds, not milliseconds. I/O crosses a virtualisation boundary and pays for the trip. Every pod carries its own guest kernel in memory - this fleet books 350 MiB of overhead against each microVM so the scheduler doesn't lie to itself about what fits. Probation-tier work is batch-shaped, not interactive.

That's fine, because it's what probation means. You don't measure a supervised shift by throughput; you measure it by what it reveals. The standard tier - shared kernel, warm caches, low latency - exists because the promotion is earned, and a reward only means something if the default is slower. The tax isn't a flaw in the ladder. The tax is the ladder.

Trust is a history, not a property

Trust is not a property of software. No scanner emits it, no licence contains it, and a clean hash only proves that today's stranger is the same stranger as yesterday - it says nothing about what the stranger does. Trust is a history: observed behaviour, under constraint, accumulated until it's boring. And it's graduated, never granted on arrival.

The reason to bother is worth saying plainly. A model is not a config value you swap on a hunch; it is the part of the system that decides what is true, and it arrives with a wider executable surface than almost anyone pictures. Weights, agents, plugins, anything that turns up executable - the ladder is the same. Start it where an escape lands in a guest kernel, where the network answers to a written declaration, where there are no credentials to spend. Let the record grow. Promote on the record, or not at all.

Give every new arrival a place to build a history where the worst it can do is bounded.

Live on the fleet's untrusted tier: Kata 3.31.0 baked into a Talos node image, RuntimeClass kata booting per-pod KVM microVMs (guest kernel 6.18.28 on a 6.18.34 host), agent-sandbox v0.4.6 wrapping them in Sandbox resources, a zero-peer default-deny namespace holding not one credential - the supervised shift the next downloaded model walks into.