Can remote storage beat recomputing cached model states?

The paper's discussion section overturns what it calls a traditional assumption about offloading KV cache to remote storage. "Traditionally, it was believed that loading KV cache from remote storage primarily improves cache hit rates and reduces storage costs... but at the cost of increased inference latency, since loading data from remote devices was thought to be slower than performing a full prefill." The authors report this tradeoff no longer holds for at least one of LMCache's users: "Company C... adopted LMCACHE to load KV caches from their own remote object store, achieving 22–32% lower TTFT compared to full prefill" — loading cached states beat recomputing them outright.

The shift traces to storage hardware rather than to LMCache's own software. The old assumption rested on "the historically low throughput of remote object stores such as Amazon S3, which offered loading speeds as low as 100 MBps." The paper notes that "with Amazon S3 Express, the throughput has increased from 100 MBps to nearly 1 GBps," and argues that once a remote backend is fast enough, it "can simultaneously improve cache hit ratios and reduce TTFT" rather than trading one for the other.

This finding is the production evidence behind the broader architecture described in Can KV cache move off GPU memory for reuse?. It also qualifies the caution in Does prompt caching actually reduce agentic LLM costs? : here, whether offloaded caching helps or hurts latency depends on the backing store's throughput, not on caching as such. The same discussion section separately reports, from "real traces from Company F," that truncating long-context inputs to keep only the most recent tokens drops prefix cache hit ratios "from roughly 85% to 45%," and cites "other studies" (Ji, 2025) making a related point about dynamically edited context invalidating prefix reuse — a phenomenon Does dynamic tool discovery break prompt caching benefits? describes for a different mechanism, changing tool schemas rather than truncating context; the excerpt does not name MCP or claim either paper updates the other.

Both figures come from single, anonymized enterprise customers (Company C, Company F), reported by LMCache's own developers rather than by an independent study. The 22–32% TTFT improvement depends on the customer's own remote object store matching or exceeding S3 Express-class throughput, a condition the excerpt does not establish as typical across deployments. The finding is best read as a case report that remote KV cache offloading's latency cost can now turn negative under favorable storage conditions, not as a general benchmark of remote-versus-local cache loading.