Serving Gemma4 with Rust on vLLM 🦀

This tutorial walks through installing and setting up the Rust toolchain for vLLM on an AWS EC2 G5g instance — Graviton2 (aarch64) with an NVIDIA T4G GPU — and getting vLLM's Rust frontend (vllm-rs) built, running, and verified. This paper is a follow-on to the original G5g Gemma 4 build. Everything below was run on the box. 🦀 Wait, vLLM has Rust in it? You betcha. Since PR #40848 (merged 2026-05-21), vLLM vendors a 14-crate Rust workspace: bench chat cmd engine-core-client llm managed-engine metrics mock-engine parser parser/python server text tokenizer tracing Edition 2024, resolver 3. Straight from the vendored rust/Cargo.toml: Crate Version Job axum 0.8.8 the HTTP server tokio 1.47.1 async runtime zeromq 0.6.0 talks to the Python engine rmp-serde / rmpv 1.3.1 msgpack on the wire minijinja 2.22 chat templates tonic / prost 0.14.6 / 0.14.3 gRPC — remember this one It's a drop-in replacement for the Python FastAPI server. Two artifacts get built: 🦀 vllm-rs — the axum frontend binary 🐍 vllm._rust_tool_parser — a PyO3 extension module Rust is a build requirement now That's the headline, and it's reason enough on its own: you cannot build vLLM from source at v0.27.2rc0 without Rust in the picture. setup.py imports it at module scope, line 21, unguarded: from setuptools_rust.build import build_rust No try, no feature flag, no opt-out. Metadata generation doesn't happen without it. And this isn't a quirk of one release. vLLM's Rust surface is 14 crates covering the HTTP frontend, the tool parser, the tokenizer and the benchmark client, and it has been growing since it landed. If you build inference infrastructure from source, a Rust toolchain is becoming table stakes — so it's worth knowing how to drive it properly rather than working around it. Three things do get conflated, though, and they have different scopes: Component Needed to build vLLM? Needed to serve? setuptools_rust (Python pkg) yes, always no cargo / rustc toolchain for working Rust artifacts no protoc for vllm-rs specifically no Then why doesn't pip install vllm need this? Because normally pip installs it for you. pyproject.toml declares it: [build-system] requires = [ "cmake>=3.26.1", "ninja", "packaging>=24.2", "setuptools>=77.0.3,=8.0", "setuptools-rust>=1.9.0", #