Two tricks fight the same memory crunch in AI models: one reorganizes storage, the other just deletes less-useful memory.

This explores two different responses to the same problem: as a language model reads and generates text, it keeps a growing store of intermediate results (the KV cache), and that store eats GPU memory. The short version is that PagedAttention changes *where* the cache is stored and keeps all of it, while block-wise pruning changes *what* is stored and drops some of it. They solve different problems and can be used together. One caveat first: this collection has a strong source on PagedAttention but only one direct example of block-wise pruning, so the comparison rests on a fairly thin base on the pruning side.

PagedAttention borrows a trick from operating systems. Normally each request's cache needs one large, contiguous chunk of memory reserved ahead of time, and about 30% of that space goes unused through fragmentation. PagedAttention splits the cache into fixed-size blocks that can sit anywhere in memory, like virtual-memory pages, and requests that share a prefix can share the same blocks Can OS paging techniques reduce LLM serving memory waste?. Nothing is deleted. The model sees exactly the context it would have seen otherwise. The payoff is throughput: less memory is wasted, so more requests fit on the GPU at once, which lifts serving speed 2–4×. It's an infrastructure fix, and the model's behavior stays the same.

Block-wise pruning works differently because it treats the cache as working memory to curate. The Thread Inference Model structures reasoning as a tree of subtasks. When a subtask finishes, rule-based pruning discards that subtask's cached blocks and keeps only its result Can recursive subtask trees overcome context window limits?. That is lossy on purpose, and its goal is different from paging's. Paging aims to fit more users on the hardware. Pruning aims to let one model reason past its context limit, and TIM stays accurate even while editing about 90% of its cache. Packing is a storage question, and pruning is a question of what to remember, which is closer to what reasoning needs.

Why is there a cache to manage at all? Transformers keep every past token available, which is exactly why they beat fixed-memory state-space models at copying and retrieving from context Can state-space models match transformers at copying and retrieval?. Pruning gives up a little of that perfect recall to save memory. Paging keeps all of it. A third option sits between them: instead of discarding evicted context, compress it into the model's weights during an offline 'sleep' phase. That research argues the real long-context bottleneck is the compute needed for this compression, not memory capacity Is long-context bottleneck really about memory or compute?.

The same choice between reorganizing memory and discarding it shows up at other layers. Prompt caching reuses identical prefixes across API calls, much as PagedAttention shares blocks between requests. It cuts agentic costs by 41–80%, but caching content that will never be reused can actually add latency Does prompt caching actually reduce agentic LLM costs?. Agent harnesses face a similar choice when they move state out of the context window into REPLs, subagents, and disk Can external state caches let models solve harder problems?. In each case the useful question is whether you need more room or less to remember. Paging gives you more room, and pruning gives you less to remember.