Two fixes for AI's memory-hogging problem: one squeezes more into GPU memory, the other reuses what's already been computed elsewhere.

This explores two different fixes for the same problem: the KV cache is too big for GPU memory. PagedAttention packs that memory more tightly on the GPU. LMCache moves it off the GPU so it can be reused later. One saves space and the other saves work, and they solve different problems.

PagedAttention borrows a trick from operating systems. Before it, each request reserved one contiguous block of GPU memory sized for the longest output it might produce. Much of that reservation went unused, and the leftover gaps were wasted. Paging stores the cache in small blocks that can sit anywhere in memory, so roughly 30% of wasted memory comes back, and requests with the same prefix can share blocks. More requests fit on the GPU at once, which is where vLLM's 2–4× throughput gain comes from Can OS paging techniques reduce LLM serving memory waste?. The goal is to fit more concurrent work into a fixed amount of memory.

LMCache asks a different question: why throw the cache away when the request ends? It pulls KV states out of GPU memory and stores them, so a later query, or even a different serving engine, can load them instead of reading the same document or conversation history from scratch. The reported gains reach 15× on multi-turn and document-heavy enterprise workloads Can KV cache move off GPU memory for reuse?. The non-obvious part is that this only works because storage got fast. Engineers long assumed that fetching cached states from remote storage would be slower than recomputing them. With object stores near 1 GB/s, one enterprise customer saw 22–32% lower time-to-first-token by loading instead of recomputing Can remote storage beat recomputing cached model states?. The memory limit turns into a storage-hierarchy question.

The two approaches stack: vLLM can page memory on the GPU and offload to LMCache at the same time. They also have something in common. Both pay off most when many requests share the same starting text, which is the idea behind commercial prompt caching. Benchmarks show prompt caching cuts agentic API costs by 41–80%, but caching content that never repeats can make latency worse Does prompt caching actually reduce agentic LLM costs?. Reuse is also fragile. Cache hits need an exact prefix match, so letting agents discover tools dynamically, which changes the tool definitions in the prompt, quietly breaks caching Does dynamic tool discovery break prompt caching benefits?.

If you want a third option, some research shrinks the cache itself rather than storing it better. One model structures reasoning as recursive subtask trees and prunes cache entries by rule, which keeps reasoning accurate beyond the context limit Can recursive subtask trees overcome context window limits?. Another line of work argues the real long-context bottleneck isn't memory capacity at all. On that view, it's the compute needed to fold old context into the model's internal state Is long-context bottleneck really about memory or compute?. Taken together: PagedAttention packs the cache better, LMCache reuses it, pruning makes it smaller, and consolidation turns it into something other than a cache.