Operating systems split memory into flexible chunks to avoid waste — does that trick backfire when it's borrowed for serving AI models?

This explores when OS-style memory paging stops helping LLM serving and starts getting in the way. To be clear up front, the corpus has strong material on why paging helps and almost nothing that directly measures where it hurts. So the answer below works from neighboring evidence rather than one study that tests this.

Start with what paging fixes. While a model generates text, it keeps a running memory of the conversation so far, called the KV cache. Older serving systems reserved one large unbroken block of GPU memory for each request, sized for the longest output that request might produce. Most of that space was never used. PagedAttention splits the KV cache into small blocks that can sit anywhere in memory, the same way an operating system manages virtual memory. It also lets requests share blocks they have in common. Together this removes about 30% of wasted memory and gives 2-4× higher throughput at the same latency Can OS paging techniques reduce LLM serving memory waste?. That points to where the benefit lives: memory waste and memory sharing. If a workload has little of either (short, predictable requests, small batches, or memory that isn't the bottleneck), there is less to gain. Paging still adds a cost, because the model has to look up where each block lives every time it reads the cache. The corpus doesn't measure that cost, but the logic of the result implies it.

The clearest lateral warning comes from a close cousin of paging: prompt caching. A benchmark across OpenAI, Anthropic, and Google found caching cuts agentic API costs by 41-80%. But caching the full context naively can make things slower, because it spends effort storing dynamic content that will never be reused Does prompt caching actually reduce agentic LLM costs?. The same lesson likely carries over to paging and block sharing. Memory management that assumes content will be reused costs you when it isn't. What matters is whether your workload actually repeats itself, more than which technique you pick.

The workload itself also keeps changing, which shifts where paging pays off. Agent designs that separate reasoning from tool results stop the prompt from growing so fast and allow steps to run in parallel Can reasoning and tool execution be truly decoupled?. That means fewer, shorter caches to manage. Going the other way, long-context systems that read 4K-token chunks instead of short snippets Can long-context models resolve retriever-reader imbalance?, or that compress documents into short gist summaries Can LLMs read long documents like humans do?, change how much cache each request needs and how much requests have in common. Routing easy queries to smaller models Can routers select the right model before generation happens? changes the request mix again. Paging is a bet on one kind of memory pressure, many requests whose lengths vary and which share prefixes, and design choices like these can raise or lower that pressure.

The takeaway you might not have expected: the useful question is less "does paging help?" and more "does my workload actually reuse and fragment memory?" The best documented harm in this neighborhood comes from optimizing for reuse that never happens, not from paging itself. If you want to measure paging's own overhead, the corpus doesn't cover it yet. That's a real gap.