How VIDRAFT Hit 510.58 TPS on Gemma-4: A Deep Dive into "The First Gemma Challenge" Win

How VIDRAFT Hit 510.58 TPS on Gemma-4: A Deep Dive into "The First Gemma Challenge" Win TL;DR: VIDRAFT topped "The First Gemma Challenge" leaderboard with a verified 510.58 tokens-per-second (TPS) score on google/gemma-4-E4B-it using a single NVIDIA A10G GPU — while a rival submission posted a faster raw number but failed the quality gate. This post breaks down the public configuration choices that made it possible, and what engineers can steal for their own inference tuning work. What it is "The First Gemma Challenge" was a constrained inference-speed competition with two hard rules: one fixed GPU (NVIDIA A10G), one fixed model (google/gemma-4-E4B-it). Participants could not swap in stronger hardware or a lighter model. The only lever available was software-level optimization. The scoring metric was TPS (Tokens Per Second), but with a simultaneous Perplexity (PPL) budget — submissions whose PPL exceeded approximately 2.42 were disqualified regardless of speed. The organizers also ran a blind re-evaluation against a held-out prompt set that participants never saw, which meant any configuration over-fitted to self-reported benchmarks would get caught. VIDRAFT's winning submission — configuration name vidraft-fw188-ctk49-n64-patchbridge-v1 — posted: TPS: 510.58 PPL: 2.3930 Status: Passed blind re-evaluation ✅ A competing entry recorded 535.91 TPS, but its PPL landed at approximately 2.44, breaching the quality threshold. That's why the lower raw number was recognized as the verified SOTA. How it works The public manifest.json for the winning configuration reveals three conceptual optimization pillars: 1. Sliding-window attention narrowing (SLIDING_WINDOW=188) The KV-cache memory bandwidth is the dominant bottleneck during autoregressive generation. Restricting the attention window to only the most recent tokens reduces this pressure and increases throughput — but shrink the window too far and you lose context, causing PPL to spike. The value 188 is notably not a round number, which strongly suggests it was determined empirically rather than chosen from a default. The team overrode the model's text_config.sliding_window via HF_OVERRIDES and enabled Flash Attention sliding (FA_SLIDING=1) to match. 2. Centroid top-k kernel tuning (CENTROID_TOP_K=49) This parameter sits closer to the kernel level and affects both throughput and PPL simultaneously. According to the source analysis, values like 44, 48, and 49 were tested sequentially — the goal being to find the highest value that still kept PPL within budget. Bigger is not automatically better; it's a Pareto search under the quality constraint. 3. Warm-up discipline — keeping initialization outside the measurement window The configuration uses: WARMUP_BRIDGE=1, WARMUP_NUM_PROMPTS=64, WARMUP_MAX_TOKENS=1, WARMUP_SEED=42. This fires 64 single-token dummy prompts before the timed benchmark begins, so that CUDA graph capture and JIT compilation costs are absorbed before the clock starts. The source article notes this warm-up was worth approximately 15 TPS — a significant margin in a competition decided by tens of TPS. Equally important: PRECACHE_BENCH=0 is explicitly set, disabling a flag that would have inflated the self-reported TPS. The team chose to measure what the blind evaluator would actually see. Other notable parameters (as published) Speculative decoding: enabled via SPECULATIVE_CONFIG, with num_speculative_tokens=7 and method=mtp — a draft-then-verify approach that increases tokens generated per forward pass MAX_MODEL_LEN=4096 GPU_MEMORY_UTILIZATION=0.90 MAX_NUM_BATCHED_TOKENS=512 MAX_NUM_SEQS=1 Benchmarks & results Submission TPS PPL Blind eval VIDRAFT (vidraft-fw188-ctk49-n64-patchbridge-v1) 510.58 2.3930 ✅ Passed Competing entry 535.91 ~2.44 ❌ Failed (PPL > 2.42) The key takeaway: raw throughput rank and validated rank diverged because the quality gate was enforced on a held-out prompt distribution, not the participants' own test sets. How to try it The model used in the competition is publicly available from Google on Hugging Face: huggingface-cli download google/gemma-4-E4B-it The specific VIDRAFT configuration (vidraft-fw188-ctk49-n64-patchbridge-v1) and any VIDRAFT-specific tooling are not confirmed as publicly released at the time of writing. Check VIDRAFT's Hugging Face organization and their GitHub for updates. If access channels are announced, they will appear there first. FAQ Q: Why does the PPL threshold matter more than raw TPS in a "speed" competition? A: Because TPS without a quality floor is trivially gamed — you can degrade output until the model produces garbage very quickly. The PPL ceiling plus blind re-evaluation together enforce that the speed number reflects real, deployable inference quality. Q: Can I apply these same techniques to other models or GPUs? A: The concepts — quality-gated parameter search, warm-up separation, sliding window tuning, speculative decoding — are general inference engineering practices. The specific numeric values (SLIDING_WINDOW=188, CENTROID_TOP_K=49, etc.) were tuned for google/gemma-4-E4B-it on a single A10G and should be treated as starting points, not copy-paste targets, for different hardware or model configurations. Q: What is speculative decoding (method=mtp) doing here? A: A smaller, faster "draft" model predicts several future tokens ahead of the main model. The main model then verifies them in a single forward pass. If the predictions are accepted, you effectively generate multiple tokens per step — boosting measured TPS without changing the model weights or degrading output quality. Originally reported by note (일본) (2026-08-15) — source article.