Why does a language model's memory of everything it's read so far get wasted more and more as conversations get longer?

This explores why the KV cache, the running memory of past tokens that a model keeps while generating text, wastes more space as sequences get longer and models get bigger. The basic mechanics are simple. For every token, at every layer, the model stores a key and a value vector. So the cache grows with sequence length times model depth and width. The waste comes from how that memory has traditionally been handed out. A serving system doesn't know how long a response will be, so it reserves one contiguous block sized for the worst case. Most of that block sits empty, and the leftover gaps between blocks can't be reused. Longer maximum lengths mean bigger reservations. Bigger models mean each reserved slot costs more. The library's clearest doorway is Can OS paging techniques reduce LLM serving memory waste?. It treats the cache the way an operating system treats RAM: small fixed-size pages stored wherever there's room, plus sharing across requests that use the same prompt. That removes roughly 30% of wasted memory and lifts throughput 2–4× at the same latency.

The surprise is that waste isn't only an allocation problem. A lot of what sits in the cache may not be worth keeping. Can models think longer by forgetting intermediate reasoning? finds that during long chains of reasoning, most intermediate tokens soon stop mattering. Keeping only the original instruction plus a recent window gives about a 3× speedup with no retraining, and it lets reasoning run past 100k tokens. Can recursive subtask trees overcome context window limits? goes further. It structures reasoning as a tree of subtasks so finished branches can be dropped from the cache, and it keeps accuracy even when 90% of the cache is being rearranged. Read together, these suggest that a growing share of a long sequence's cache is dead weight. Paging fixes how the memory is laid out. Pruning asks whether it should be stored at all.

The obvious question is why we don't use a fixed-size memory instead. The corpus shows the cost. Can state-space models match transformers at copying and retrieval? proves that models with a fixed-size internal state can't reliably copy or look up arbitrary earlier content, while transformers can, precisely because they keep every token around. The ever-growing cache is the price of exact recall. Can neural memory modules scale language models beyond attention limits? offers a middle path. Attention handles recent context, and a learned memory compresses older material, prioritizing surprising tokens. It scales past 2 million tokens without the cost of full attention over everything.

Two neighbouring notes push back on calling this purely a memory problem. Is long-context bottleneck really about memory or compute? argues the real limit is the compute needed to fold evicted context into the model's internal state, not storage itself. On small hardware, Does recomputing weights cost less than moving them on mobile? shows that moving data can cost more than recomputing it. That hints that the size of the cache may matter less than how often it has to be moved.

A caveat: the corpus has one paper directly on KV cache allocation. It doesn't include a detailed breakdown of how waste scales with model size, so the scaling explanation above is standard background rather than a finding from these notes. What it does offer is a useful way to see the problem: there's an allocation fix (paging), a fix for what's worth remembering (pruning), and an architectural fix (compressed memory). Each gives up something different.