Saving an AI's 'memory' to disk instead of recomputing it can be faster — but only if your storage is quick enough to make it worthwhile.

This explores when it's faster to keep a model's already-computed work (its KV cache) in storage and load it back than to redo the computation from scratch. The corpus has one direct data point and a few useful neighbors. It doesn't have a systematic study of storage hardware requirements, so the conditions below are pieced together from those sources. The direct evidence comes from LMCache: one enterprise customer cut time-to-first-token by 22–32% by pulling cached states from remote object storage instead of re-running the prefill step Can remote storage beat recomputing cached model states?. The key condition is throughput. Modern object stores like S3 Express deliver roughly 1 GBps. That speed overturned the long-standing assumption that remote loading would always lose to recomputation.

So the main condition is that transfer time (cache size divided by storage bandwidth, plus fixed lookup overhead) has to come in under the time the GPU would spend recomputing. That makes this a moving threshold, not a fixed spec. Faster storage, larger caches that are expensive to recompute, and slower or busier compute all favor offloading. Run the same calculation in reverse and recomputation wins. MobileLLM is the mirror case: on phones, moving weights through memory is so slow that running the same transformer block twice beats fetching a separate set of weights Does recomputing weights cost less than moving them on mobile?. Taken together, the two notes give one rule. Wherever data movement is the bottleneck, recompute. Wherever compute is the bottleneck and storage bandwidth is high, offload.

Bandwidth alone isn't enough. What you store and how you lay it out also matter. A benchmark of prompt caching across OpenAI, Anthropic, and Google found that caching improves time-to-first-token by 13–31% in agent workflows. It also found that naively caching the whole context can *increase* latency, because it stores dynamic content that never gets reused Does prompt caching actually reduce agentic LLM costs?. So the reuse rate is itself a condition: offloading only pays if the stored state actually gets read again. On the memory-layout side, PagedAttention borrows virtual-memory paging from operating systems to store KV blocks in non-adjacent chunks and share them across requests. This removes about 30% of wasted memory and raises throughput 2–4× Can OS paging techniques reduce LLM serving memory waste?. Block-structured caches like this are also what make it practical to move pieces of state in and out of slower storage.

More broadly, offloading is one level of a storage hierarchy. Prime Agent treats an agent's state as four tiers: weights, context, a persistent workspace with subagents, and disk-backed history. Each tier trades speed for capacity Can external state caches let models solve harder problems?. The design question is which state belongs on which tier, given how often it's needed. A related note suggests that for very long contexts, the real bottleneck may not be storage at all. It may be the compute needed to turn old context back into something the model can use Is long-context bottleneck really about memory or compute?. Fast storage helps only if the reload step stays cheap.

The main takeaway is that "is offloading worth it?" isn't a property of the storage device alone. It's a ratio between storage bandwidth, compute speed, cache size, and reuse rate, and that ratio has shifted as object stores got faster. On cloud GPUs it now favors offloading. On phones it favors recomputing.