Why do agents that trim old chat or shorten tool results end up paying full price again on every AI call?

This explores why tidying up an agent's context, by dropping old turns or swapping long tool outputs for summaries, can quietly undo prompt caching. The short version is that caches reuse only an exact, unchanged prefix. The corpus has no paper that tests pruning against caching directly. It does have enough pieces to explain the mechanism and to show how some systems get around it.

The key fact comes from work on tool discovery. A cache hit needs the start of the prompt to match exactly, token for token, what was cached before Does dynamic tool discovery break prompt caching benefits?. The model's internal state for each token depends on every token that came before it. So if you change something early in the prompt, every cached state after that point is stale. Pruning a middle turn or replacing a 5,000-token tool result with a three-line summary edits the prompt in exactly the place a cache can't tolerate. The cache stops at the first changed token, and the system has to recompute everything after it. The same paper's fix also applies here: keep a fixed, stable core at the front of the prompt and push variable material later, where changes cost less.

The stakes are large. Across OpenAI, Anthropic, and Google, caching cut agentic API costs by 41–80% and sped up the first token of each reply by 13–31% Does prompt caching actually reduce agentic LLM costs?. The same benchmark found that caching everything blindly can actually increase latency, because content that will never repeat still gets written to the cache. Taken together, the two papers show that pruning and caching are both bets about which parts of the context stay fixed. They conflict when you make different bets for each.

The more interesting part is how other designs handle this. Compression methods like LLMLingua rewrite the prompt itself. They can shrink it 20x with little accuracy loss Can prompts be compressed without losing model performance?, but a rewritten prompt is a new prefix, so you trade cache reuse for fewer tokens. ReWOO-style planning takes a different route: it plans the reasoning up front and keeps tool observations out of the main reasoning flow, so the prompt doesn't keep growing and changing with every tool call Can reasoning and tool execution be truly decoupled?. Recursive Language Models go further and never put the long prompt in context at all. They store it in a code environment and query it when needed Can models treat long prompts as external code environments?.

The surprise is that you can prune the cache itself instead of the text. The Thread Inference Model structures reasoning as a tree of subtasks and uses fixed rules to drop finished branches directly from the KV cache (the model's stored internal state for earlier tokens). It keeps reasoning accurately even after changing 90% of that cache Can recursive subtask trees overcome context window limits?. Related work argues that the real long-context bottleneck isn't memory but the compute needed to turn discarded context into lasting internal state Is long-context bottleneck really about memory or compute?. That points to a different framing: the problem isn't pruning as such. It's pruning at the text level, after the cache has already been built around the original text.