“no kernel image is available for execution on the device” What This PyTorch Error Means On RunPod Blackwell GPUs

You launch a training job on RunPod. PyTorch detects your GPU, torch.cuda.is_available() returns True, and your environment checks appear to pass. Then your script reaches its first real CUDA operation and crashes: RuntimeError: CUDA error: no kernel image is available for execution on the device If you are running on a Blackwell GPU with an older PyTorch build, you have hit a GPU architecture compatibility problem. The frustrating part is that your environment looks perfectly healthy right up until the first CUDA kernel executes. What causes this error CUDA kernels are compiled for specific GPU architectures, identified by compute capability for example sm_86 for an A40, sm_90 for an H100. When you install PyTorch, the build you download contains kernels compiled for a fixed range of architectures. PyTorch 2.4.1+cu124 supports sm_50 through sm_90. Blackwell GPUs: the RTX PRO 4000, RTX PRO 4500, RTX PRO 6000, and RTX 5090 have compute capability sm_120. They are outside that range. With an incompatible PyTorch build, PyTorch does not fail immediately. torch.cuda.is_available() returns True. The GPU name appears in the output. Everything looks fine. Then the training script reaches the first real CUDA kernel call the first matrix multiplication, the first forward pass and it throws the error above. By that point the pod is already allocated, your dependencies are installed, and billing has started. Note on the B200: The B200 belongs to the sm_100 architecture family, not sm_120. Always check the actual compute capability reported by your GPU rather than assuming every Blackwell card uses the same architecture. You can check this in two lines: import torch cap = torch.cuda.get_device_capability(0) print(f"Device capability: sm_{cap[0]}{cap[1]}") print(f"Supported architectures: {torch.cuda.get_arch_list()}") If your device capability is not in the supported list, the run will fail when the first kernel executes. Why torch.cuda.is_available() returning True is not enough This is where the problem becomes deceptive. torch.cuda.is_available() confirms that PyTorch can access CUDA and identify a device. It does not confirm that the installed build contains compatible kernels for every operation you intend to run. Architecture compatibility is a separate check. GPU detection and GPU compatibility are two different things. First-hand reproduction on RunPod I reproduced this on a RunPod RTX PRO 4000 Blackwell pod using the default PyTorch 2.4.1+cu124 template. The terminal output looked clean: ✓ PyTorch 2.4.1+cu124 detected (CUDA available — NVIDIA RTX PRO 4000 Blackwell) Meanwhile PyTorch was printing a warning to stderr that sm_120 was not supported by this build. The pre-flight check passed. Training would have failed on the first real kernel call. The warning that appeared in stderr: UserWarning: CUDA device capability sm_120 is not supported by this PyTorch build. Supported capabilities: sm_50 through sm_90. If your training script runs in a tmux session, pipes output to a log file, or uses a framework like Axolotl or Unsloth, you will miss this warning entirely until the forward pass crashes. The pod also reported /workspace - 466126.9 GB free of 2190892.5 GB unusually large volume figures that appear to be a virtualisation reporting artefact on early Blackwell deployments rather than real capacity. This reproduction exposed a gap in the original pre-flight checks and led directly to the GPU architecture check added in ComputeFence v0.3.2. Which GPUs are affected with standard PyTorch images As of PyTorch 2.4.x, these cards are not supported by the builds shipped in standard RunPod PyTorch templates: NVIDIA RTX PRO 4000 Blackwell: sm_120 NVIDIA RTX PRO 4500 Blackwell: sm_120 NVIDIA RTX PRO 6000 Blackwell: sm_120 NVIDIA RTX 5090: sm_120 NVIDIA B200: sm_100 (different architecture, same class of problem) If RunPod assigns one of these GPUs and you are on the default template, you will hit this error on the first kernel call. How ComputeFence catches it Since v0.3.2, ComputeFence compares each GPU’s compute capability against the architectures supported by the installed PyTorch build using get_device_capability() and get_arch_list(). A mismatch is flagged as a blocker not a warning: ✗ BLOCKER: GPU architecture mismatch NVIDIA RTX PRO 4000 Blackwell has compute capability sm_120 This PyTorch build supports sm_50 through sm_90 Training will fail on the first CUDA kernel call Fix: Use a PyTorch build compiled for sm_120, or select a different GPU This runs before billing starts. uvx computefence doctor Note: get_arch_list() reports architectures compiled into the PyTorch installation. It is a useful compatibility signal but does not guarantee that every third-party CUDA extension will work. Run a small representative CUDA operation before committing to a long training run on new hardware. How to fix it Option 1: Install a PyTorch build that supports your GPU Use a release compiled for sm_120. Check pytorch.org/get-started/locally for current compatibility. After installing, verify with the two-line check above before launching your job. Option 2: Select a different GPU If you need to use an existing PyTorch image with pinned dependencies, select a GPU whose architecture is in the supported range. An A40 (sm_86) or H100 (sm_90) will work with PyTorch 2.4.x. The broader lesson GPU detection is not GPU compatibility. As providers add newer hardware generations, the gap between the latest cards and the PyTorch builds in standard templates will keep appearing. Passing torch.cuda.is_available() is not proof that your training workload can execute. Check the architecture before billing starts. GitHub: github.com/Francisco-Booth/ComputeFence Install: pip install computefence