Do Drafters Work on Small Models? Gemma 4 E2B and E4B on Two Small GPUs

This article provides a step by step guide to serving Gemma 4 E2B and E4B with Google's QAT assistant drafters through llama.cpp's speculative decoding, and measuring what the drafter is worth on two small NVIDIA cards: a desktop RTX 3050 OEM and a laptop GTX 1650 Ti. A suite of Python MCP tools is built to simplify management of the llama.cpp hosted deployment. This is the second part of a short series. Part 1, Gemma 4's MTP Drafter, Explained, covers how speculative decoding and Gemma 4's multi-token prediction (MTP) drafter work and how the exact Q4_0 repack is built. This part has the measurements. https://github.com/xbill9/gemma4-dev A drafter is the guesser in speculative decoding: it proposes the next few tokens, and the main model checks them all in one pass. Do drafters work on small models? Yes. Gemma 4 E2B and E4B are already small, and their drafter is a 4-layer model of 59 MB. Served one request at a time, it makes both models faster on both cards, and GSM8K and BFCL scores stay the same: one request at a time RTX 3050 GTX 1650 Ti E4B 1.72x 1.49x E2B 1.40x 1.18x Answer quality unchanged unchanged The smaller model gains less. E2B is already fast per token, so each drafted step costs a larger share of what it saves (inferred from the draft-length results below, not measured directly). Both cards keep the same share of the drafted tokens, within 0.03, so the gap between the cards comes from the cards. The gain has limits. With several requests at once it shrinks on both cards, and on the 1650 Ti it turns into a loss from 4 requests (E4B) or 8 (E2B). On the 1650 Ti, E2B's gain is gone by about 32k tokens of conversation. Too many drafted tokens also cost speed on the slower card. The drafter here is an exact Q4_0 repack of Google's weights. It matches the converter's Q8_0 file on speed and answers at 60% of its size, and beats Google's bf16 weights on speed on every card and model. What Is Being Measured? A language model writes one token per forward pass, and every pass reads every weight. For one user on a small GPU, each pass spends most of its time moving weights, which leaves compute to spare. Speculative decoding uses it: a small drafter guesses the next few tokens, and the main model checks all of them in one pass. It keeps the guesses up to the first one it disagrees with and adds one token of its own, so the output is the main model's either way. When most guesses are kept, one pass yields several tokens instead of one. Gemma 4's drafter uses multi-token prediction (MTP). It is a 4-layer model that reads the main model's hidden state and reuses its KV cache, so it keeps no cache of its own and costs little more than its weights. MTP is one of several ways to do speculative decoding. llama.cpp 77c4540ce offers ten through --spec-type: a separate small draft model (draft-simple), draft-eagle3, draft-mtp, draft-dflash, draft-dspark, and five n-gram methods. MTP is the one Google builds drafters for: E2B, E4B, 12B, 26B-A4B and 31B each have a published assistant checkpoint (E2B's config names it Gemma4AssistantForCausalLM), which llama.cpp runs as draft-mtp and vLLM as "method":"mtp". This article measures MTP only. Part 1 explains this in full. Google publishes an assistant drafter for each Gemma 4 QAT checkpoint. Both cards here serve the exact Q4_0 GGUFs of gemma-4-E2B-it and gemma-4-E4B-it with the drafters stored back as Q4_0 on their trained 4-bit grid, 59 MB each. The question is what the drafter is worth on each card, and why the two cards differ. At This Point You Should Have… An NVIDIA GPU with a CUDA driver — here an RTX 3050 OEM (driver 615.78.08) and a GTX 1650 Ti with Max-Q Design (driver 615.71.09) The CUDA toolkit, 13.4 here, and cmake llama.cpp at a commit with --spec-type draft-mtp — both cards ran 77c4540ce for every comparison below Python 3.10 or newer, the system python3, no virtualenv The repository cloned, with the four rigs local-llamacpp-{3050oem,1650ti}-{2b,4b}-q4_0/ Claude Code, or any MCP client that speaks stdio The Two Cards RTX 3050 OEM GTX 1650 Ti Max-Q Host i7-5820K desktop, 46 GiB i7-10750H laptop (Lenovo Yoga 9), 15 GiB Memory 8192 MiB 4096 MiB Architecture Ampere, compute capability 8.6, tensor cores Turing TU117, compute capability 7.5, no tensor cores Power limit 120 W about 40 W E2B, no drafter 108.2 tok/s 76.5 tok/s E4B, no drafter 58.3 tok/s 38.0 tok/s Without a drafter the 3050 generates 1.41x faster on E2B and 1.53x on E4B. Different hosts, so treat ratios between the cards as indicative. Step 1 — Repack the Drafter on Its Trained Grid Google publishes the QAT assistant drafters as google/gemma-4-E2B-it-qat-q4_0-unquantized-assistant and google/gemma-4-E4B-it-qat-q4_0-unquantized-assistant. Each is a 4-layer model that reads the main model's KV cache, stored as bf16 safetensors. Quantization-aware training already put the weights on a 4-bit grid: in every group of 32 values there are at most 16 distinct levels, evenly spaced. bf16 spends 16 bits storing each of those levels. llama.cpp's convert_hf_to_gguf.py has no way to know that, so it writes Q8_0: 8.5 bits per weight, re-rounded onto an 8-bit grid of its own. The file shrinks, and the values move. Only 11.7% of the drafter's weights come back bit-identical. Q4_0 is 4.5 bits per weight: one fp16 step per group of 32, and a 4-bit level for each value. That is enough to hold the trained grid, provided each group's step is read off its own values. gguf_exact.py does this for each group: amax = np.abs(g).max(-1, keepdims=True) for m in range(1, 9): # try steps amax/1 .. amax/8 cand = amax / m q = np.clip(np.rint(g / cand), -8, 7) good = np.abs(g - q * cand).max(-1, keepdims=True)