Can KV cache move off GPU memory for reuse?

KV cache — the stored intermediate attention states a model uses to avoid recomputing a prompt's earlier tokens — has traditionally lived only in GPU memory, tied to a single query. The paper's central claim is that this no longer scales: "over time, the total size of KV cache that is stored by the users continues to grow, and it far exceeds the capacity of the GPU memory." LMCache is presented as "the first and so far the most efficient open-source KV caching solution," extracting KV cache from inference engines so it can be "shared... across engines and queries." Combined with vLLM, the authors report "up to 15× improvement in throughput" on workloads such as multi-round question answering and document analysis.

The mechanism rests on three contributions described in the excerpt. First, optimized data movement: batched, pipelined transfer at a "configurable chunk size" larger than the engine's native page size, zero-copy operations between storage tiers, and layer-wise pipelining that overlaps GPU compute with cache loading. Second, a modular "KV cache connector" interface that decouples LMCache from inference engines that change quickly — the paper notes "15–20 new open-weight models are released every week in 2025," forcing engines like vLLM and SGLang to keep altering their internal KV cache layout. Third, a control API — "pinning, lookup, cleanup, movement, and compression" — across "GPU, CPU, storage, and network layers." The paper frames the result as treating KV cache "as a first-class data structure rather than an internal byproduct of inference."

Where Can OS paging techniques reduce LLM serving memory waste? solves memory waste inside GPU memory through page-table-style allocation, LMCache addresses what happens once cache needs to leave the GPU entirely — for reuse by a later query, or transfer to a different GPU during prefill-decode disaggregation. It sits underneath the cost and latency effects measured in Does prompt caching actually reduce agentic LLM costs?: that benchmark measures caching's effect at the API-pricing level, while LMCache is open-source infrastructure enterprises are reportedly using to realize such effects at the serving-stack level. It also bears on Does dynamic tool discovery break prompt caching benefits?: both describe prefix-cache reuse breaking when context changes, though LMCache's own discussion locates this in context truncation rather than tool-schema churn.

The 15× figure is a ceiling ("up to"), drawn from workloads the paper's own authors — who are also LMCache's developers — selected for evaluation; the excerpt does not report typical or worst-case gains, or independent benchmarking outside that evaluation. The system also targets only two inference engines, vLLM and SGLang, so how the approach generalizes to other serving stacks is untested here. The implication, held to the strength of the evidence, is that enterprises running high-volume, multi-turn or document-heavy LLM workloads now have a concrete lever — cache offloading across storage tiers — whose upper-bound benefit this paper's own authors report, with its typical benefit under more varied workloads still to be shown by outside evaluation.