Can caching a prompt's repeated opening both speed up answers and get reused more often, or does one gain cost the other?

This explores whether prefix caching, which means storing the already-processed start of a prompt so later requests can skip that work, can reuse more of the cache and also make the first token arrive sooner. The short answer from the corpus is yes, usually, but only if you choose carefully what goes into the cache. The clearest evidence comes from a benchmark across OpenAI, Anthropic, and Google agent workloads. Caching cut API costs by 41–80% and improved time-to-first-token by 13–31% in multi-turn agentic tasks Does prompt caching actually reduce agentic LLM costs?. In practice, then, the two goals usually move together: when the model reuses more of its earlier work, it has less to process before it can start answering.

The same study points to where they can pull apart. Caching the whole context naively can actually *increase* latency, because it spends effort storing changing content, such as tool outputs, timestamps, and the latest user turn, that will never be reused. So the conflict isn't really between hit ratio and speed. It's between caching everything and caching what stays stable. The practical lesson is to put the stable parts (system prompt, tool definitions, long reference documents) at the front and push the changing parts to the end. This raises the share of each request that matches the cache and avoids paying to cache content that is used only once.

A 115-day case study of a long-running agent shows how far this can go. 82.9% of all tokens were cache reads, which is high enough that the authors argue cost should be measured per finished piece of work rather than per token Do persistent agents really cost less per token?. That kind of hit ratio only happens when the context is built to persist and grow by adding to the end, not by being rewritten.

This is where the corpus points to a less obvious tension. Methods that shrink context also save time and money: LLMLingua compresses prompts up to 20x Can prompts be compressed without losing model performance?, and ACON cuts an agent's peak token usage by 26–54% Can prompt optimization cut agent token usage without retraining?. But each time compression rewrites earlier context, the prefix changes, and the cached work for that prefix can no longer be used. This is an inference from how prefix caches work; neither paper measures it directly. Compression and caching are two different ways to avoid reprocessing tokens, and combining them carelessly can cancel out the gains. The same idea of sharing a prefix also appears outside serving: in RL training, branching many attempts from one shared prefix produces more distinct attempts for the same token budget Can shared-prefix trees reduce redundancy in agent rollouts?.

What the corpus lacks is the systems side: cache eviction policies, request scheduling that groups shared prefixes together, or measurements of how hit ratio and first-token latency trade off under heavy load on a shared server. If you want that layer, these notes won't take you all the way. They do make clear that the main lever is how you order the prompt, not a cache setting.