Loading a saved AI 'memory' from storage used to be slower than just rebuilding it — faster cloud storage is now flipping that math.

This explores how the speed of remote storage changes a basic serving tradeoff. When a model reads a prompt, it builds a KV cache, the internal states it uses to keep generating. If the same document or conversation prefix comes back later, you can rebuild that cache from scratch (prefill) or fetch a saved copy. For years the working assumption was that fetching from remote storage was too slow, so recomputing won. The corpus suggests that assumption no longer holds. Object stores like S3 Express now reach roughly 1 GB per second, and one enterprise deployment cut time-to-first-token by 22–32% by loading cached states remotely instead of recomputing them Can remote storage beat recomputing cached model states?.

The mechanism is simple. Loading time grows with cache size divided by storage throughput. Prefill time grows with prompt length and the cost of running the model. Once the pipe is fast enough, moving bytes beats doing math. This matters most where the same long context keeps coming back: multi-turn chats and document-heavy enterprise work. In those settings, systems like LMCache that pull the KV cache out of GPU memory, so it can be reused across queries and even across separate serving engines, report throughput gains of up to 15× Can KV cache move off GPU memory for reuse?. Fast remote storage turns the cache from a temporary scratchpad into something shared and long-lived.

The interesting twist is that the same move-versus-recompute calculation can flip the other way on different hardware. On phones, where memory bandwidth is the choke point, MobileLLM found it cheaper to run a transformer block twice than to fetch a second set of weights Does recomputing weights cost less than moving them on mobile?. So the lesson isn't that caching always wins. Whichever is slower, moving data or computing it, sets the design, and storage speed is what moves that line. PagedAttention is the earlier, on-GPU version of the same idea: it borrowed operating-system memory paging to stop wasting cache space, which made the cache cheap enough to share between requests at all Can OS paging techniques reduce LLM serving memory waste?.

There's also a caution. Faster loading only pays off if the cached content actually gets reused. A benchmark of commercial prompt caching found 13–31% faster time-to-first-token in agentic tasks, but caching everything without discrimination can add latency by storing changing content that never comes back Does prompt caching actually reduce agentic LLM costs?. Where reuse is heavy, the economics change: a 115-day study of a persistent agent found 82.9% of its tokens were cache reads Do persistent agents really cost less per token?. At that level, storage throughput is a main factor in cost and speed, not a side detail.

The corpus has little on the finer points, such as how latency changes with cache size, network distance, or compression. It doesn't offer a precise formula. What it does show is that an old rule of thumb flipped once storage got fast enough. That shift is a big part of why sharing KV caches across machines and sessions is now practical.