Init-gating GPU readiness on Kubernetes

The most common way a GPU workload fails at the edge isn't the model, the driver, or the network. It's timing. Kubernetes is eager - it will happily schedule your inference pod the moment a node reports Ready, which is often before the NVIDIA device plugin has advertised nvidia.com/gpu. The pod starts, can't see a GPU, crash-loops, and now your rollout is poisoned across the fleet, on boxes nobody is standing next to.

Node-Ready answers the wrong question. It says the kubelet is up. It says nothing about whether the one piece of hardware your workload exists to use is actually there yet. So stop trusting it: make GPU readiness explicit, and gate on that.

Gate the schedule, then gate the start

The first gate is free - a resource request. A pod that requests a GPU won't schedule until the device plugin advertises capacity:

resources:
  limits:
    nvidia.com/gpu: 1

That handles the common case. But on a single-GPU edge node recovering from a power cut, there's a window where the plugin has advertised the device and the driver is still finding its feet - and you don't want an expensive model load to be the thing that discovers it. So the second gate is an init container that blocks until the device is demonstrably real, and fails loudly if it never is:

#!/usr/bin/env bash
set -euo pipefail
# Block until the GPU is visible AND healthy, or fail loudly after a bound.
for i in $(seq 1 30); do
  if nvidia-smi -L | grep -q '^GPU 0'; then
    echo "GPU ready"; exit 0
  fi
  echo "waiting for GPU ($i/30)"; sleep 5
done
echo "GPU never became ready" >&2
exit 1

Two gates, two failure modes closed: the scheduler can't place the pod before capacity exists, and the workload can't start before the hardware answers. Note the bound - an init gate that waits forever isn't a gate, it's a hang. Two and a half minutes, then fail loud and let the platform retry. Fail-closed, never fail-quiet.

Why this is the win

Once readiness is gated, the entire class of "pod started before the GPU" failures disappears, and it disappears the same way on every node. That consistency is the real prize at the edge. A fix that requires a human to notice, shell in, and nurse a bad rollout doesn't scale past the first dozen sites; a gate that makes every node converge identically after every reboot does.

Design the dependency, don't hope for it

At the edge, design the dependency - don't hope for it. Anything your workload cannot run without deserves an explicit, bounded, fail-loud gate between it and the scheduler's optimism. The GPU is just the first dependency worth naming; egress paths and model artifacts are next, and they want the same treatment.