Gemma 4 on an Old 4 GB Laptop GPU: QAT Takes It From 9.5 GiB to 1.6

This article provides a step by step deployment guide for Gemma 4 E2B's quantization-aware-trained (QAT) checkpoint to a local, laptop hosted GPU enabled system — a much older Lenovo Yoga 9 with a 4 GB GTX 1650 Ti. A suite of Python MCP tools is built to simplify management of the llama.cpp hosted deployment. https://github.com/xbill9/gemma4-dev/tree/main/local-llamacpp-1650ti-2b-q4_0 Gemma 4 E2B in bfloat16 is 9.5 GiB of weights. This laptop's GPU has 4 GiB. Even a plain int8 conversion would not fit. The QAT release closes that gap: the same model, trained knowing it would be stored at 4 bits, in a 3.35 GB file of which only 1.31 GiB ever has to be on the GPU. It loaded in 1618 MiB, left more than half the card free, and decodes at 73.75 tok/s. What is this project trying to Do? Every other rig in this repository serves Gemma 4 from rented hardware: Cloud TPU, Compute Engine, EC2, Cloud Run. I wanted to know whether the same model would run on the laptop I already had — a 10th-generation Core i7 with a GTX 16-series GPU, no tensor cores and a 4 GB ceiling that no quota request can raise. It does, and QAT is the reason. The rest of this article is how, and what the old hardware changes about running it. At This Point You Should Have… An NVIDIA GPU with a CUDA driver — here a GTX 1650 Ti with Max-Q Design, driver 615.71.09 The CUDA toolkit — nvcc 13.3 here Python 3.10 or newer for the MCP server — 3.14.7 here, the system python3, no virtualenv The repository cloned, and local-llamacpp-1650ti-2b-q4_0/ as your working directory Claude Code, or any MCP client that speaks stdio The Laptop Machine Lenovo Yoga 9 15IMH5 CPU Intel Core i7-10750H, 10th generation (Comet Lake), 12 threads RAM 15 GiB, as free reports it GPU GeForce GTX 1650 Ti with Max-Q Design, 4096 MiB GPU power 40 W limit The MCP server reports the GPU the same way: gpu_status 📡 **GPU** — `local-llamacpp-1650ti-2b-q4_0` NVIDIA GeForce GTX 1650 Ti with Max-Q Design, 7.5, 4096 MiB, 1606 MiB, 2127 MiB, 615.71.09 ⚠️ GTX 16-series (TU116/TU117): compute capability 7.5 but **no tensor cores**. Do not compare throughput against the T4-based `g4dn`/`g5g` rigs on the strength of a matching compute capability. Compute capability 7.5 is a trap. A T4 is also 7.5 and has tensor cores; the GTX 16-series is a cut-down Turing die with the tensor cores removed. llama.cpp notices on its own at init: The following devices will have suboptimal performance due to a lack of tensor cores: Device 0: NVIDIA GeForce GTX 1650 Ti with Max-Q Design Why Can't the Normal Model Fit? The sizes, from this repository's model reference: Gemma 4 E2B as Weights Fits 4096 MiB? bfloat16 9.5 GiB ❌ no int8 ~4.8 GiB ❌ no QAT Q4_0 GGUF, whole file 3.35 GB barely, on paper QAT Q4_0 GGUF, GPU-resident part 1.31 GiB ✅ yes The bf16 checkpoint is more than twice the card. Halving it to int8 is still larger than the card. Only a 4-bit model is in range, and that is where QAT comes in. What Is QAT, and Why Does It Matter Here? Quantization-aware training simulates quantization during training rather than compressing a finished model afterwards. The 4-bit weights are not a post-hoc approximation of a bf16 model: the model was trained knowing it would be stored this way. Google's model card puts the goal plainly: This model card is for the new versions of the Gemma 4 family optimized with Quantization-Aware Training (QAT), which allows preserving similar quality to bfloat16 while dramatically reducing the memory requirements to load the model. That is the difference that matters on a 4 GB card. Rounding the bf16 model down to 4 bits after the fact would fit too, but nothing in its training would have prepared it for that. QAT is the version of "fits" that was designed to keep the model's quality close to bf16 on the way there. Google ships the QAT weights four ways: Artifact What it is For -qat-q4_0-unquantized QAT values in a half-precision container custom compilation -qat-q4_0-gguf the same values, packed Q4_0 llama.cpp — this rig -qat-w4a16-ct compressed-tensors vLLM mobile on-device runtimes phones The GGUF is not a lesser copy. This repository checked it: four norm tensors read out of the GGUF are bit-identical to the ones in the -unquantized release, so it is the same QAT model in its native packing. Which QAT Checkpoint? google/gemma-4-E2B-it-qat-q4_0-gguf, which llama.cpp opens directly. The agent reads it off disk: model_info 📡 **Model** — `local-llamacpp-1650ti-2b-q4_0` - **Name:** `google/gemma-4-E2B-it-qat-q4_0-gguf` - **Path:** `/home/xbill/models/gemma-4-E2B-it-qat-q4_0/gemma-4-E2B_q4_0-it.gguf` - **On disk:** 3.35 GB - **Quantization slot:** `q4_0` — but the dominant tensor type is **Q6_K**. Both embedding tensors are Q6_K (2.257 GB of 3.334 GB); only the ~1.08 GB transformer body is actually Q4_0. - **Resident on GPU:** ~1.31 GiB. `per_layer_token_embd` (1.93 GB, 58% of the file) is `TENSOR_READ_LAZY` and is served by GET_ROWS out of the mmap. Run `inspect_gguf.py` to re-derive the split from the artifact rather than trusting these numbers. Two lines in that output explain why a 3.35 GB file fits in far less than 3.35 GB. Why Only 1.31 GiB Has to Be on the GPU A 3.35 GB file against 3724 MiB free reads as "barely fits." It is better than that. make info reads the tensor table out of the GGUF: make info gemma-4-E2B_q4_0-it.gguf tensors: 541 total: 3.334 GB largest tensors: 1926.8 MB per_layer_token_embd.weight Q6_K [8960, 262144] /dev/null initialize OK: name='local-llamacpp-1650ti-2b-q4_0' version='' proto 2025-06-18 tools/list OK: 7 tools -> get_help, gpu_status, model_info, model_server_status, query_model, start_model_server, stop_model_server One Thing Broke: MCP SDK 2.x Midway through, the server stopped loading. Another project on the machine needed the MCP Python SDK 2.x, all of these projects share one system Python, and 2.x renamed the class this server imports: python3 -c "from mcp.server.fastmcp import FastMCP" raise ModuleNotFoundError(_MESSAGE, name=__name__) ModuleNotFoundError: No module named 'mcp.server.fastmcp'. This is mcp 2.x, where FastMCP was renamed to MCPServer (from mcp.server.mcpserver import MCPServer) and other APIs changed; see the migration guide at https://py.sdk.modelcontextprotocol.io/v2/migration/#fastmcp-renamed-to-mcpserver or pin 'mcp=1.2.0,=2 -sys.modules["mcp.server.fastmcp"] = _fastmcp_module +sys.modules["mcp.server.mcpserver"] = _mcpserver_module Touching start_model_server for the rename surfaced an older bug: the tool launched llama-server without -fa, -t or --parallel, so an MCP-started server came up with 4 slots and 6 threads instead of the measured configuration — and llama.cpp splits the context across slots. The argv now lives in one function, a test holds it to the Makefile, and the running server's real command line matches: tr '\0' ' ' < /proc/$(pidof llama-server)/cmdline /home/xbill/llama.cpp/build/bin/llama-server -m /home/xbill/models/gemma-4-E2B-it-qat-q4_0/gemma-4-E2B_q4_0-it.gguf --host 127.0.0.1 --port 8080 -ngl 99 -c 8192 -ctk f16 -ctv f16 -fa 1 -t 4 --parallel 1 --metrics make lint make test All checks passed! lint OK ---------------------------------------------------------------------- Ran 30 tests in 0.035s OK And Price/Performance? There is no price. The laptop was already on the desk: nothing is billed, nothing is reserved, and there is no idle cost to optimise. What QAT bought is capability, not a discount. Without it this GPU cannot hold Gemma 4 E2B at all. With it, the model takes 1618 MiB and a single conversation decodes at 73.75 tok/s. Teardown # make serve is in the foreground: Ctrl-C Or ask the agent for stop_model_server, which sends SIGTERM to whichever process owns port 8080, including one started by make serve. Not run for this article; the server is still up. What This Does Not Cover Output quality was not measured here. The claim that QAT holds quality close to bfloat16 is Google's, from the model card; this article measured memory and speed, not accuracy, and never compared the QAT model's answers against a bf16 run. No benchmark completed a task. Every sweep generation hit its token cap inside Gemma 4's thinking block, so the throughput figures are real and the tasks are not. Nothing here transfers to a T4. Same compute capability, different silicon. Summary The goal of this article was to run Gemma 4 on a much older laptop whose GPU has 4 GB of memory. The key to the solution was Google's quantization-aware-trained checkpoint, packed as a GGUF, plus llama.cpp leaving its largest tensor in host memory. The measured results were: bf16 needs 9.5 GiB and int8 ~4.8 GiB; the QAT GGUF needs 1.31 GiB on the GPU. Only the 4-bit QAT model fits this card, and it fits with room to spare. Full offload in 1618 MiB of 4096, with an 8192-token context and a KV cache of just 60 MiB. 73.75 tok/s single-stream decode on a GPU with no tensor cores; flash attention is worth 4.8%. Quantizing further is a loss: a q8_0 KV cache costs 11–12% of decode, because QAT already took the memory pressure away. An MCP server manages the whole lifecycle, and moving it to SDK 2.x took a two-line change that also exposed an MCP-started server running the wrong configuration. Scope: one Lenovo Yoga 9 15IMH5 (Core i7-10750H, 15 GiB RAM, GTX 1650 Ti Max-Q with 4096 MiB and a 40 W cap), llama.cpp 95ef7fc built for sm_75 with CUDA 13.3, google/gemma-4-E2B-it-qat-q4_0-gguf, and the MCP server on mcp 2.2.0 under Python 3.14.7. llama-bench rows are three repeats; serving sweeps are three repeats per level with the prompt cache defeated. The memory split comes from a sibling rig running the same engine through Ollama on the same card. Model quality was not measured. The strategy for using MCP for a local GPU deployment was validated with an incremental step by step approach. References local-llamacpp-1650ti-2b-q4_0 | GitHub google/gemma-4-E2B-it-qat-q4_0-gguf | Hugging Face Quantization-Aware Training for Gemma 4 | Google llama.cpp | GitHub Migration Guide: v1 to v2 | MCP Python SDK