When a chatbot's memory of your conversation gets too big for the GPU, what's faster: storing it elsewhere, or just making it think again?

This explores what happens to response speed when a model's KV cache (the saved intermediate state that lets it skip reprocessing text it has already seen) leaves scarce GPU memory for cheaper, slower storage. The core tradeoff is a race between two costs: the time to *move* saved state back onto the GPU, and the time to *recompute* it from the raw prompt (called prefill). The corpus doesn't have fine-grained measurements of GPU-to-CPU transfer over the hardware link itself. What it does have is a clear picture of when moving beats recomputing, and why that answer keeps changing.

The surprising finding is that the old intuition, that recomputing is always faster than fetching from far away, no longer holds. LMCache reports an enterprise customer getting 22–32% lower time-to-first-token by loading cached state from *remote* object storage instead of recomputing it, because modern stores like S3 Express now move data at roughly 1 GBps Can remote storage beat recomputing cached model states?. Once state can live off the GPU, it can also be shared across queries and even across separate serving engines. That's where LMCache's reported throughput gains of up to 15× on multi-turn and document-heavy workloads come from Can KV cache move off GPU memory for reuse?. Note the distinction: offloading mostly buys *throughput and reuse*. Lower latency only shows up when the cached content actually gets reused.

That reuse condition is where things go wrong. A benchmark of prompt caching across OpenAI, Anthropic, and Google found 13–31% faster time-to-first-token in agentic tasks, but naive full-context caching can actually *increase* latency, because you pay to store and manage dynamic content that will never be read again Does prompt caching actually reduce agentic LLM costs?. The lesson carries over to GPU-to-CPU offloading: moving state costs time on every transfer, so it only pays off if you're moving the right things.

The opposite direction is also worth seeing. On memory-starved mobile hardware, MobileLLM found it faster to *recompute* a transformer block twice than to fetch a second set of weights Does recomputing weights cost less than moving them on mobile?. That's the same move-vs-recompute question with the opposite answer, because there the data path is the bottleneck. Together these show the tradeoff isn't fixed: it depends on how fast your storage link is compared to your compute. Before offloading anything, PagedAttention showed that a lot of GPU memory pressure is simply waste. Splitting the KV cache into non-contiguous pages recovers about 30% of memory lost to fragmentation and lifts throughput 2–4× at the same latency Can OS paging techniques reduce LLM serving memory waste?. So it's often worth managing memory better on the GPU before moving anything off it.

Some work reframes the question entirely. One line argues the real long-context bottleneck isn't storage capacity but the compute needed to turn evicted context into something the model can use Is long-context bottleneck really about memory or compute?. Another prunes the KV cache selectively during reasoning, so less state needs to be kept or moved in the first place Can recursive subtask trees overcome context window limits?. The takeaway: offloading is one of three levers, alongside storing more efficiently and keeping less, and the fastest systems tend to pull all three.