Why vLLM and SGLang might not be the best option to run AI agents

vLLM and SGLang are great inference engines. If you serve chat or batch jobs, use them. But AI agents are a different workload, and I wanted to see how well these engines handle it. So I measured it. How an agent talks to the model A coding agent works in turns: It sends the whole conversation to the model. The model answers with a tool call (run a command, edit a file). The agent runs the tool, appends the result, and sends the whole conversation again. In the traces I used, a median session has about 60 turns and grows to tens of thousands of tokens. Every turn re-sends everything before it. To answer a turn, the engine needs the KV cache for the full history. If the cache from the last turn is still in GPU memory, only the new tokens need computing. If it was evicted while the agent was running its tool, the engine computes the whole history again. What vLLM and SGLang do between turns Both engines have a prefix cache: blocks from earlier requests stay in memory and get reused if the next prompt starts the same way. But they don't know that a request belongs to an agent that will come back in half a second. vLLM 0.33.0 accepts a session_id on chat requests, but it's only used as metadata for KV cache events. There's no pinning and no time-to-live. When memory runs out, blocks are evicted least recently used first. SGLang 0.5.23 has real sessions (/open_session, /close_session), but its OpenAI-compatible chat endpoint only forwards the session_id, not the session parameters. The mode that keeps a session's KV pinned (--enable-streaming-session) is off by default and needs the /generate API. Eviction is LRU by default, with options like TLRU. So under load, while one agent waits on its tool, other agents' requests push its history out of memory. When it comes back, the engine pays for that history again. The test I wrote a small serving engine in Rust (about 1,700 lines) that keeps each agent's session in GPU memory between turns, shares the common prompt start (system prompt, tool definitions) across sessions, and only admits new sessions when the ones already running still fit. I compared it with vLLM and SGLang: Workload: 24 real coding-agent sessions from the SWE-rebench-OpenHands dataset (791 turns), all running at once. Same conditions for every engine: Qwen3-8B, one NVIDIA L40S, and the same KV cache size (173,200 tokens). Tool time: a fixed wait between turns of 0.5 s, 5 s or 60 s. Measured coding-agent tool calls have a median gap of about 0.42 s, so 0.5 s is the realistic one. Two runs per configuration. I wrote down before each run what result would count as a pass. Results Total time for all 24 sessions with a 0.5 s tool wait (lower is better): Engine Run 1 Run 2 History recomputed First token (median) vLLM 0.33.0 2264 s 2222 s 55% 28.6 s SGLang 0.5.23 (LRU) 2013 s 2048 s 51% 35.0 s SGLang 0.5.23 (TLRU) 2067 s 2024 s 52% 36.3 s Rust session engine 1506 s 1586 s 8-16% 4.5 s More than half of all the prompt tokens vLLM and SGLang processed were history they had already computed once and then evicted. Keeping sessions in memory cut that to 8-16% and made the whole run 1.31x faster than SGLang and 1.45x faster than vLLM. The first token came about 5x sooner. The gain shrinks as tools get slower: Tool wait vs best baseline vs vLLM 0.5 s 1.31x faster 1.45x faster 5 s 1.13x faster 1.24x faster 60 s about the same about the same With a 60 s wait, few agents need the GPU at the same time, so there's less memory pressure and every engine does about as well. My engine was also slower to the first token there. What this doesn't mean vLLM and SGLang aren't slow. Token for token, they're faster than my engine: its decoding is 15-21% slower per token than vLLM. The gain comes only from not recomputing history. The idea isn't new. InferCept, Pensieve and Continuum, among others, already proposed keeping agent KV caches alive across tool calls. What I added is a measurement against the current versions of both engines on real agent traces. It's one setup. One model, one GPU, one load level, fixed tool waits. I didn't test CPU offload (LMCache, SGLang HiCache) or SGLang's streaming sessions, which might close part of the gap. Not everything worked. Three of my pinning policies lost to vLLM with 60 s waits. Pinning sessions for too long blocks memory other agents could use. The paper lists them all. So should you switch? Probably not yet. My engine is a research prototype, and the right fix is for vLLM and SGLang to learn about agent sessions. Both projects have open proposals for exactly that: vLLM PR #38514 and RFC #37003, and SGLang RFC #24656. But if you run many agents on your own GPUs, check how much of your prompt traffic is recomputed history. Both engines can report cached tokens per request (--enable-prompt-tokens-details in vLLM, --enable-cache-report in SGLang). If it's over half, like in my test, you're paying for the same tokens twice. Code and paper Everything is open: the engine, the replay client, the raw results and the scripts that produce every number above. Code: https://github.com/Nahuel990/agent-session-engine Paper: When Does Keeping Agent Sessions in GPU Memory Pay Off? A Rust Serving Engine and a Replay Study on Real Agent Traces (in the repo under paper/)