LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference

KV cache has traditionally been stored in GPU memory to accelerate the decoding phase of large language model (LLM) inference. However, it is increasingly necessary to move KV caches outside GPU devices, to enable cache reuse across different queries, and inference engines. Our real-world usage statistics confirm this trend: over time, the total KV cache stored by users has grown rapidly, far exceeding the capacity of GPU memory. Despite this need, there lacks an efficient solution for offloading and transferring KV caches. We present LMCACHE, the first and so far the most efficient open-source KV caching solution, which extracts and stores KV caches generated by modern LLM engines (vLLM and SGLang) out of the GPU memory and shares them across engines and queries. LMCACHE supports both cache offloading (prefix reuse across queries) and prefill–decode (PD) disaggregation (cross-engine/GPU cache transfer). LMCACHE’s high performance and wide adoption stem from the following contributions: (i) highly optimized KV cache data movement powered by batched data movement operations, compute and I/O pipelining; (ii) a modular KV cache connector component, decoupling LMCACHE from the rapid evolution of inference engines; (iii) a first-class control API, such as pinning, lookup, cleanup, movement, and compression, for flexible cache orchestration across GPU, CPU, storage, and network layers. Our evaluation shows that combining LMCACHE with vLLM achieves up to 15× improvement in throughput across workloads such as multi-round question answering and document analysis. Large-scale adoption of LMCACHE in enterprise settings provide us valuable insights, for example, fetching KV cache from remote storage has unsurprisingly benefits to prefill delay, and that context truncation, which is a widely applied technique in industry, can greatly reduce prefix cache hit ratio by half. The source code of LMCACHE is at: https://github.com/LMCache/LMCache.

Introduction. Today, large-language model (LLM) inference has outpaced training in growth. LLM inference powers millions of applications, from interactive customer support and code generation to retrieval-based document analysis and agentic workflows. To build the systems for LLM inference, KV cache, the intermediate states of large language model (LLM) inference, has now become a de facto optimization to make inference faster. Traditionally, KV cache has been used during new token generation to skip re-computing the KV cache of the input prompts, within a single query. Thus, every query is processed independently by one instance of the inference engine, and a LLM query’s lifecycle, including computation and I/O operations, happens in the GPUs and the GPU memory of one inference engine. However, recent trends has advanced to propose moving KV cache outside the GPU memory, including two emerging directions:

Cross-query caching to avoid redundant compute: KV cache can be persisted (e.g., to lower-tier storage devices) beyond the lifecycle of a query to avoid any re-computation of the shared prefix for another query.

Prefill–decode disaggregation for higher utilization: There is an arising trend to decouple the prefill and decode phase on different GPUs, to ensure that latency-sensitive decoding phase is not affected by throughput-oriented prefill phase.

Such prefill-decode (PD) diaggregation requires KV cache produced by the prefill GPUs to be transferred to the decode GPUs. Our real-world usage statistics confirm the trend of moving KV cache out from the GPU memory. As will be discussed in detail soon (§2.2), over time, the total size of KV cache that is stored by the users continues to grow, and it far exceeds the capacity of the GPU memory. This suggests that KV cache may need to be frequently evicted from GPU memory as more and more requests come in. Also, in order for it to be reused by another query, it needs to be offloaded from GPU memory, and then loaded back to GPU memory for reuse. To make these ideas practical, the LLM inference systems must be augmented with new KV cache semantics. In particular, inference engines should support the new interface that extracts KV caches from a normal inference call and re-loads KV caches into subsequent queries on demand. The system also must allow the extracted KV caches to be stored persistently and transferred across distributed inference engines. Most importantly, for such interface extensions to be practical, KV cache extraction, re-loading, storage, and transfer must be efficient, and the new interfaces must remain compatible with rapidly evolving inference engines such as vLLM Kwon et al. (2023) and SGLang Zheng et al. (2024). We introduce LMCACHE, the first open-source library that provides a high-performance implementation of these new KV cache semantics. With LMCACHE, KV cache can be extracted from and loaded back to inference engines efficiently, stored in a hierarchy of storage devices (CPU memory, local disk, remote disk, and Redis), and transferred over different networks (Ethernet, RDMA, NVLink). LMCACHE makes three distinct contributions.

#1. Highly optimized performance: LMCACHE incorporates a series of performance optimizations that make storing and loading KV cache efficient and practical in real deployments. For instance, LMCACHE batches operations to pipeline the storing and loading of KV cache, as well as to pipeline GPU compute and data loading/storing (e.g., loading next layer’s KV cache while performing computations for the current layer). Moreover, rather than storing/loading KV cache at the granularity of the inference engine’s native small page size, LMCACHE stores/loads KV cache at a configurable chunk size, often much larger than page size, to fully utilize the bandwidth between storage devices and GPU memory. LMCACHE also minimizes the copies of KV cache data when moving them among different storage tiers, by implementing zero-copy operations.

#2. Standardized interface with inference engines: LM- CACHE defines standardized connector interfaces that remain compatible with fast-changing inference engine backends. On average, 15–20 new open-weight models are released every week in 2025, so to best utilize new hardware for the new models, modern LLM inference engines must evolve rapidly, potentially changing the KV cache layout in GPU memory and thus affecting the LMCACHE interface. To address this, LMCACHE designs and implements a modular KV cache connector interface that decouples LMCACHE with the inference engine backend, so LMCACHE can easily adapt to the evolving APIs in the inference engines.

#3. Flexible KV cache management interface: The interface augmentation introduced by LMCACHE exposes KV cache, a new data structure in LLM inference. LMCACHE exposes APIs that allow developers and operators to locate, move, pin, and even compress KV cache extracted from inference engines.

Related work. 3.2 Related Work and Existing Solutions Several KV cache handling mechanisms exist, but none of them fully address the above challenges:

Inference frameworks: Since the release of vLLM Production Stack vLLM project (2025) in January 2025, there have been several open-source distributed inference stacks, including Nvidia’s Dynamo NVIDIA Corporation (2025), AIBrix Team et al. (2025), llm-d llm-d Project (2025), SGLang OME Team (2025), and KServe Contributors (2025b). They focus on easy deployment of inference engine solutions over Kubernetes, and technically, they all support various query routers based on load or prefix cache awareness and support KV caching, where LMCACHE is used in vLLM production stack, Dynamo, llm-d, and KServe.

Inference engine-native KV caching: Open-source inference engines, like vLLM and SGLang, also offer native GPUto-CPU KV cache transfers, but it is designed for single-node inference, so they lack cross-node transfer optimization or hierarchical storage support for KV cache. We will evaluate their performance and compare it against LMCACHE in §8.

KV cache storage layers: Mooncake Qin et al. (2025b), Redis Redis (2025), InfiniStore ByteDance (2025), and 3FS Contributors (2025a) provide distributed object storage or caching, but they lack an efficient “glue” layer between the inference engines to efficiently move small tensors frequently across different storage tiers, or are tightly coupled with a specific inference framework.

Proprietary implementations: Proprietary inference APIs (e.g., Fireworks AI, Together AI) implement their own prefix caching internally, but these are tied to their closed-source serving stacks and are not accessible to operators deploying their own infrastructure.

Source code for research: Several research proposals have open-sourced prototypes for their KV cache optimizations, including prefix caching Chen et al. (2025); Gao et al. (2024); Gim et al. (2024); Jin et al. (2024, 2025); Kwon et al. (2023); Lee et al. (2024); Yang et al. (2025); Ye et al. (2024); Yu et al. (2025); Zhao et al. (2024); Zheng et al. (2024), PD disaggregation Patel et al. (2024); Shi et al. (2025); Zhong et al. (2024), and KV cache compression Du et al. (2025); Ge et al. (2024); Jegou et al. (2024); Li et al. (2025, 2024); Liu et al. (2024b); Qin et al. (2025c); Tang et al. (2024); Xiao et al. (2024a,b); Zhang et al. (2025). However, these prototypes are typically built on research-oriented inference framworks, such as HuggingFace Transformers, not fully enterprise-ready, or are not designed to evolve alongside the rapidly changing inference engine ecosystem, such as SGLang and vLLM.

Method. 4 Overview of LMCACHE LMCACHE addresses these challenges by a unified, highperformance KV caching layer capable of efficient storage, movement, and explicit management of KV caches for pagedmemory inference engines, making prefix caching and PD disaggregation practical at enterprise scale. As a KV caching layer, LMCACHE sits between LLM inference engines and heterogeneous storage/network devices (Figure 5). Its goal is to provide a standardized, high-performance substrate for KV cache movement and management, while remaining compatible with rapidly evolving inference frameworks such as vLLM and SGLang. Figure 6 shows the end-to-end system. Below, we walk through two example workflows: storing and retrieving KV cache.

Store: When a new query arrives, it first passes through the KV connector, which prepares metadata such as the tokenized input prompt and GPU memory addresses of the relevant pages. The query then goes to the token processor, which determines how many new tokens are not yet in the backend and need to be stored. Finally, the storage manager saves the KV cache for these new tokens to the backend via the transfer channel, which handles the data transfer logic.

Retrieve: When a query requires loading KV cache from the backend, it also starts with the KV connector to prepare metadata. The token processor identifies the number of prefixmatched tokens already in the backend. Next, the event manager checks if the same query ID has been seen before. If so, the cached memory addresses are already tracked and can be returned directly to the GPU connector, which loads the KV cache back into GPU memory. The event manager also launches asynchronous, layer-wise loading events as described in §5.2. If the query ID is new, it is forwarded to the storage manager to look up the CPU memory addresses of the stored KV cache.

Lookup: When a query needs to check whether the KV cache for specific tokens exists in the backend, higher-level components such as routers query the cache controller. The cache controller maintains a token pool that records all tokens currently stored in the KV cache backend. Whenever a LM- CACHE instance stores or evicts a KV cache, the LMCACHE worker inside the instance updates the token pool with the new status. This ensures the token pool always has the up-to-date information of tokens in the backend.

5 Performance Optimizations An important aspect of LMCACHE is improving the efficiency of KV cache movement across devices. In enterprise-scale LLM inference, LMCACHE addresses three key challenges:

To address the I/O inefficiency caused by small KV cache units, LMCACHE introduces a set of optimizations.

Configurable Chunk Size: Rather than transferring KV cache at the page level, LMCACHE groups multiple pages from multiple layers into larger chunks, with a default size of Parallel Store/load Operations: LMCACHE supports parallel storage and retrieval of KV caches across multiple storage tiers, including local CPU DRAM or disks, remote CPU DRAM or disks, and object storage (e.g., S3). In practical LLM serving workloads, KV caches often need to be migrated across devices concurrently—for instance, transferring the KV cache of a hot context from GPU to CPU memory while simultaneously offloading a cold context from CPU memory to local disk. To maximize link utilization, LMCACHE ’s store and load APIs accept multiple source and destination devices, enabling concurrent data movement across heterogeneous links. Moreover, these operations can be executed in parallel when the interconnect supports full-duplex communication (e.g., PCIe).

5.2 Compute-I/O Overlapping LMCACHE employs multiple optimizations aiming for overlapping LLM inference computations with I/O to maximize GPU utilization.

Layer-wise pipelining: LMCACHE overlaps KV cache transfers with inference computation through layer-wise pipelining. Specifically, it assigns separate CUDA streams for inference computation and data movement within each layer. For example, before performing inference on the first layer, its KV cache is loaded into the GPU buffer and transformed into pages.

Discussion. Loading from remote storage is faster than prefill: Traditionally, it was believed that loading KV cache from remote storage primarily improves cache hit rates and reduces storage costs by leveraging cheaper storage devices, but at the cost of increased inference latency, since loading data from remote devices was thought to be slower than performing a full prefill. This assumption was largely due to the historically low throughput of remote object stores such as Amazon S3, which offered loading speeds as low as 100 MBps. Recently, however, there has been a major boost in remote storage performance, for example, with Amazon S3 Express, the throughput has increased from 100 MBps to nearly 1 GBps. Our users, such as Company C, have adopted LMCACHE to load KV caches from their own remote object store, achieving 22–32% lower TTFT compared to full prefill. This insight suggests that remote backends can simultaneously improve cache hit ratios and reduce TTFT.

Context truncation lowers prefix cache hit rates: Many industry users employ a sliding-window mechanism to handle long-context inputs constrained by limited model context windows or GPU memory. For instance, when input tokens exceed the context window limit, some companies truncate the input to keep only the most recent tokens. However, this approach significantly reduces prefix cache hit ratios, since truncated inputs no longer match the prefixes of previously cached contexts. In practice, using real traces from Company F, we observe that prefix cache hit ratios drop from roughly 85% to 45% when truncating input contexts to keep only the latest tokens. Other studies have also discussed this phenomenon, emphasizing that dynamically adding or removing context tokens should be avoided, as it invalidates prefix KV cache reuse Ji (2025).

Containerized code is preferred: With the growing scale of LLM inference, most production environments now rely on Kubernetes to manage GPU clusters. Consequently, deploying inference engines (e.g., vLLM or SGLang) and LMCACHE through containerized environments, typically via Docker images, has become the standard practice among industry users. Interestingly, many users rely solely on the official Docker images without going deep into LMCACHE ’s source code.

Unexpected high cache hit rate in production systems: Our customers did not expect such a high prefix cache hit ratio, such as 50% hit rate for company G in their production environments, until they deploy LMCACHE inside their systems. Previously, people thought KV caches could only be reused for fixed system prompts. However, modern applications increasingly exhibit "dynamically reusable contexts", such as conversation histories in coding assistants, chat applications, and retrieval-augmented generation (RAG) pipelines. These emerging patterns have significantly increased overall cache hit ratios in real-world deployments.

Industry vs. academia users: LMCACHE was designed as a unified prototype framework in May 2024, where we can put our research works into it to gain more impact. However, we found that at that time, industry wants a highly efficient KV cache offloading solution as the size of KV cache and concurrent users keep growing. Then, our attention has been moved towards improving the performance, stability, and compatibility. Since most companies are less concerned with customizing attention algorithms, we deprioritized designing flexible APIs for integrating specialized attention mechanisms, such as selective token dropping. This makes LMCACHE less popular in academia, since the academia users often focuses on modifying attention mechanisms for their research prototypes.

Conclusion. and Outlook This paper presented LMCACHE, the first open-source and most widely adopted production-ready KV caching layer for enterprise-scale LLM inference. By treating KV cache as a first-class data structure rather than an internal byproduct of inference, LMCACHE transforms LLM engines from isolated token processors into a distributed ecosystem of compute and storage. Evaluation across diverse workloads and models demonstrates that LMCACHE consistently delivers significant throughput improvement and latency reduction compared to both open-source baselines and commercial inference APIs. Beyond performance, LMCACHE has already seen rapid adoption in production environments, where enterprises leverage its CPU offloading, hierarchical storage, and PD disaggregation capabilities to keep low latency and reduce cost in trillion-token–scale deployments. Real-world deployments have also revealed new opportunities, such as KV cache reuse in recommendation systems and lossy compression in openended chatbots, underscoring the versatility of LMCACHE across application domains.