Day 3: Hardware Acceleration in Docker — GPU Passthrough and Limits for Constrained Hardware

A leading reason AI workloads fail in containers is the boundary between the container's isolated namespace and the host's hardware. Running inference on raw CPUs creates big latency bottlenecks, and exposing a GPU or accelerator takes deliberate host setup. The acceleration architecture A container cannot talk to a PCIe device without driver support. The NVIDIA Container Toolkit supplies it by wrapping the standard runtime with a hardware-aware one (nvidia-container-runtime). Diagram: FIG 1 · The NVIDIA Container Toolkit injects the host's driver libraries and device nodes into the container at launch. Your image does not bundle the kernel driver. See the animated version. Three ideas make it work. Diagram: FIG 2 · Decoupling the driver from the image is what makes accelerated containers portable. Compatibility between the host driver and your image's CUDA version still matters. See the animated version. Driver decoupling. The host runs the kernel driver (nvidia.ko). The image never contains it. Runtime injection. At launch the toolkit injects the matching user-space driver library (libcuda.so) and the device nodes (/dev/nvidia*) into the container. Application agility. The image needs only the high-level CUDA runtime libraries, so it does not depend on the host's driver build. The host driver must still be recent enough for the CUDA version inside the image. Launching an accelerated container With the toolkit installed, register it with Docker and restart the daemon. sudo nvidia-ctk runtime configure --runtime=docker sudo systemctl restart docker That edits /etc/docker/daemon.json. You can also set it by hand, and making it the default runtime is optional, since --gpus selects it per container. { "default-runtime": "nvidia", "runtimes": { "nvidia": { "path": "nvidia-container-runtime", "runtimeArgs": [] } } } docker run -d --name local-inference \ --gpus all \ --shm-size="2g" \ -p 8000:8000 \ enterprise-model-server:latest Verify it: docker run --rm --gpus all ubuntu nvidia-smi should print the same GPU table as the host. Why --shm-size matters PyTorch passes tensors between worker processes through shared memory (/dev/shm). Docker's default is only 64 MB, so parallel data loading can crash suddenly with bus errors. Raising it, for example to 2g, avoids that. Diagram: FIG 4 · PyTorch uses shared memory to pass tensors between worker processes. Docker's 64 MB default is too small for that, so raise it with --shm-size. See the animated version. Hard limits for a 16 GB machine An unconstrained container is a danger to the host. A multi-step prompt can grow the KV-cache and demand several extra gigabytes of RAM at once, so enforce limits with Linux cgroups. docker run -d --name constrained-model \ --cpus="4.0" \ --memory="10g" \ --memory-swap="10g" \ -p 11434:11434 \ ollama/ollama:latest Diagram: FIG 3 · An unconstrained model container can take the whole machine down. Kernel cgroup limits keep the damage inside the container. The 10 GB cap is an example for a 16 GB host. See the animated version. --memory="10g" caps the container at 10 GB and leaves about 6 GB for the host OS, networking daemons and file cache. --memory-swap="10g" (equal to --memory) disables swap for the container. Swapping model weights to disk slows inference by orders of magnitude, so it is better to fail fast. When the container exceeds its cap, the kernel's OOM killer stops that container, not the host. This is the default behaviour (--oom-kill-disable=false), so you do not need to set it. Never use --oom-kill-disable=true without a memory limit, because then a spike can hang the whole machine. Add --gpus only when a GPU is present. On a CPU-only laptop, the memory and CPU caps are what protect you. Treat these flags as part of your deployment definition, and in Kubernetes you will express the same limits as resource requests and limits. Run it yourself This is the exact example from the video. These are real docker run commands that read the limits from inside the container. There is no GPU on my laptop, so only the limits half of the lesson is shown. # no GPU on this machine, so limits only docker run --rm --cpus=2 --memory=256m --memory-swap=256m \ --shm-size=2g alpine:3.20 sh -c ' cd /sys/fs/cgroup for f in memory.max memory.swap.max cpu.max; do echo "$f: $(cat $f)" done echo shm: $(df -h /dev/shm | tail -1 | tr -s " " | cut -d" " -f2)' docker run --rm --memory=64m --memory-swap=64m alpine:3.20 \ sh -c 'x=$(dd if=/dev/zero bs=1M count=200 | tr "\0" x)' echo "over the cap: exit code $?" Run it: bash limits.sh Output I got while recording the video: memory.max: 268435456 memory.swap.max: 0 cpu.max: 200000 100000 shm: 2.0G over the cap: exit code 137 Coming Up Next Day 4: Optimising model weights: quantization basics with GGUF and AWQ. HardwareAcceleration #Docker #GPUs #DevOps #Linux #TechInfrastructure #CUDA #SystemAdministration Originally published at https://sureshpallapothu.in/blog/ai-int-day-3-gpu-passthrough, where this post includes animated diagrams.