Apps should dial the switchboard, never the model

This is my homelab: single-node Kubernetes clusters, one operator, a couple of consumer GPUs, and nothing here describes an employer's systems.

The first local model you stand up gets called the obvious way. The application wants an endpoint, the model server has one, so the address goes straight into config. It works immediately, which is exactly why it is a trap.

The model layer refuses to sit still. The GPU gets borrowed for a transcode run. A second backend comes online on another machine. A heavier runtime starts looking attractive for the agent workloads. Every one of those is now a change to every consumer that ever wrote the address down.

Telephones started out point to point, a wire for every pair that wanted to talk, and the exchange fixed that: every phone gets one wire, to the board, and calls are patched across the back. The model layer wants the same inversion. An application should never dial a model. It dials a switchboard.

Give every caller one number

The switchboard here is LiteLLM: a small OpenAI-compatible gateway on a virtual address pinned in the cluster's load-balancer pool. Behind it sit the model servers. In front of it sits everything that needs a completion.

The routing lives in one file. A model name appears once per backend that can serve it, and the router settings decide what happens when one of them stops answering. Trimmed from the lab's config, with example addresses:

model_list:
  - model_name: qwen2.5-7b
    litellm_params:
      model: ollama_chat/qwen2.5:7b
      api_base: http://ollama.llm-serving.svc.cluster.local:11434
  - model_name: qwen2.5-7b
    litellm_params:
      model: ollama_chat/qwen2.5:7b
      api_base: http://10.0.0.13:11434
router_settings:
  routing_strategy: simple-shuffle
  num_retries: 2
  allowed_fails: 2
  cooldown_time: 30
# two backends, one public name; a failing backend is retried, then rested 30s

Two entries, one name: a consumer asks for qwen2.5-7b and does not need to know which machine answered. That file is versioned in git, because a routing table is the operator's directory, not application config.

Unpatch the line before you stop it

The test came the week one of the cards changed jobs. That GPU also feeds an AV1 transcode queue, and giving the encode run the whole card meant stopping the model server on it. The change went in through the repository the cluster's reconciler watches, not as a command against the cluster, where the reconciler would have put it straight back:

# kubernetes/apps/llm-serving/ollama.yaml: the card goes to the transcode queue
spec:
  replicas: 0

What I recorded is that consumers dialling the gateway were not edited or re-pointed and carried on working against the other backend. I did not measure latency across the change or watch what happened to a response already streaming from the server as it stopped, and a stable address is no promise that such a call survives. Treat graceful failover as unverified until that has been tested.

There is a bill for doing it carelessly. With the settings above, a dead entry stays in the shuffle: a request that draws it pays a connect timeout and a retry before another backend answers, the entry is rested for thirty seconds after two failures, and then it comes back and does it again. That reads to a user as "the assistant has gone slow", with nothing in the logs to explain it. So the order matters: take the entry out of the routing table, let the gateway pick that up and check calls are landing on the other line, and only then stop the server.

Growth works the same way in reverse. A vLLM backend would be another line into the back of the board rather than a migration. I have not deployed one, and I have no benchmark of my own to quote for it; speaking the same API is also not the same as matching on streaming or tool calls, so it would get the same tests as any new line. The retired hand-tuned rig with the heavier models is preserved as config and could come back as a third line the same way.

Concentrate the policy at the board

Once every call passes one board, model-access policy stops being scattered and starts being placed. Each consumer gets its own scoped key carrying a model allowlist, a budget and an expiry. That only covers every request if nothing can reach a model server around the board, so treat gateway-only access as a requirement to enforce at the network, not a property the gateway grants itself.

Listing models with a key shows what it claims to allow, not what it refuses. The check that means something asks for a completion it should get and one it should not, with the consumer's own key typed in rather than written into a file or the shell history:

GATEWAY=http://10.0.0.10:4000               # lab example address
ALLOWED=qwen2.5-7b                          # in this key's allowlist
DENIED=qwen2.5-coder-7b                     # routed by the gateway, not in this key's allowlist
read -rsp 'consumer key: ' CONSUMER_KEY; echo
export GATEWAY ALLOWED DENIED CONSUMER_KEY
python3 - <<'EOF'
import json
import os
import urllib.error
import urllib.request


def call(path, body=None):
    req = urllib.request.Request(
        os.environ["GATEWAY"] + path,
        data=json.dumps(body).encode() if body else None,
        headers={"Authorization": "Bearer " + os.environ["CONSUMER_KEY"],
                 "Content-Type": "application/json"})
    try:
        with urllib.request.urlopen(req, timeout=60) as r:
            return r.status
    except urllib.error.HTTPError as e:
        return e.code


ask = {"max_tokens": 1, "messages": [{"role": "user", "content": "ping"}]}
got = {m: call("/v1/chat/completions", {**ask, "model": os.environ[m]})
       for m in ("ALLOWED", "DENIED")}
print(got)
raise SystemExit(0 if got["ALLOWED"] == 200 and 400 <= got["DENIED"] < 500 else 1)
EOF
# exit 0 only if the allowed model answers and the other is refused

The exact refusal code depends on the gateway version and its configuration; mine answered 401 when I last recorded it. Two traps sit around those lines: adding a model to the routing table is only half the job until it is in the consumer's allowlist, and the gateway image ships no curl at all, so a curl test run inside it reports a perfectly healthy gateway as dead.

There is a trust dimension as well. The agents that call this board never hold a downstream credential of their own; that argument has its own post and does not need re-running here.

Pay for the single point of failure on purpose

Every call passes the board, so if the board is down, everyone is down. That is a cost accepted deliberately, and it is affordable because the board is small: a routing table and a key list, not gigabytes of weights or a queue of half-finished work. In my lab it gets a memory reservation and a critical priority class. That is my choice for a component everything depends on, not a default to hand every gateway.

A redeployed process is not restored policy, though. Every virtual key, allowlist, budget and spend row lives in the gateway's Postgres, and that is the part worth protecting properly. The nightly file backup copies the database's files while Postgres is running: an ordinary live copy, which is neither the frozen, consistent snapshot PostgreSQL documents as a valid method nor a dump. The lab's method is a logical dump, scheduled to run ahead of the file backup. Starting first is not the same as finishing first, so the backup should only take a dump that exited cleanly and is fresh, and only a restore proves any of it. There is no restore of this one in my records yet.

The other half of the bargain is the number itself. The gateway's address is pinned in the load-balancer pool and owned by the cluster rather than by any one machine, so hardware behind it can be renumbered or rebuilt without consumers noticing. A stable DNS name would do the same job; what matters is that somebody owns the number and keeps it.

The principle

None of this is about models. Any capability consumed by many applications and served by infrastructure that changes deserves a switchboard: one stable address, one dialect, and a routing table that is somebody's deliberate, versioned decision rather than a constant that escaped into application config.

The moment two things call one service directly, you owe them a number that outlives it.

From the lab's GPU tier: one gateway on a pinned virtual address, model servers patched in and out behind it, and no gateway consumer re-pointed the week a card was handed to the transcode queue.