Ollama, llama.cpp or vLLM: pick by workload, not by benchmark Tags：llm, localllm, performance, ai

Every few weeks a new benchmark says one local inference stack is X% faster than another. The numbers are usually real. They're also usually answering a different question than the one you have. Here's how I choose, by workload. One person, trying models → Ollama. If you're pulling a new model every few days to see how it behaves, setup time is the bottleneck, not tokens per second. Ollama's one-command pull and sensible defaults win here, and a slower decode rarely matters when you're reading the output yourself. One catch: set the context length explicitly. If you compare models on defaults, you're not comparing like with like. One machine, squeeze it → llama.cpp. When the hardware is fixed (a single consumer GPU, a laptop with no discrete GPU, Apple Silicon) and you want every token per second, build llama.cpp yourself with the backend flags for your hardware. Generic prebuilt binaries often take conservative paths. You also get direct control over layer offloading, KV cache quantization and batch sizes. Many users at once → vLLM. The moment several requests arrive concurrently, the question changes from "how fast is one answer" to "how many answers per GPU-hour". Continuous batching lets finished requests leave and new ones join mid-batch, and paged KV cache keeps memory from fragmenting. Single-user benchmarks don't show this at all. How to run a fair comparison • Same model file, same quantization • Same context length, set explicitly • Same prompts, including one long prompt (prefill) and one long answer (decode) • Measure time-to-first-token and decode tokens/s separately • Test at concurrency 1 and at your real expected concurrency • Write down driver, backend and build flags If two stacks only differ under one of these conditions, that's your real answer. Short version: Ollama optimizes your time. llama.cpp optimizes one machine. vLLM optimizes a shared GPU. Pick the one that matches what you're actually short of. What does your setup look like, and where did you see the biggest gap? Post your numbers with hardware and context length below.