Does prompt caching still save money if your agent's system prompt or tool list changes size?
Do cache benefits hold with different system prompt sizes and tool counts?
This explores whether prompt caching keeps paying off as the fixed front part of an agent's prompt changes: a longer or shorter system prompt, or more or fewer tools defined for the agent.
This explores whether prompt caching keeps paying off as the fixed front part of an agent's prompt changes: a longer or shorter system prompt, or more or fewer tools defined for the agent. The short answer is that the corpus has no study that varies system prompt length or tool count step by step. What it does show is that the size of the prompt matters less than whether it stays the same from one call to the next. Caching only pays off when the beginning of the prompt is identical across calls.
The baseline numbers come from a benchmark across OpenAI, Anthropic, and Google. In multi-turn agent tasks, caching cut API costs by 41–80% and made the first token arrive 13–31% faster Does prompt caching actually reduce agentic LLM costs?. Two details are worth noticing. First, the savings vary a lot, which suggests they depend on how much of each prompt is stable and how much changes. Second, caching everything can make an agent slower, because the system spends work storing content that is never reused. So the useful question is how much of the prompt stays fixed, not how big it is. Cache the stable part, such as instructions and tool definitions, and leave out the parts that change every turn.
The number of tools matters in a less obvious way. When an agent discovers tools as it runs, as with MCP-style tool discovery, each new tool changes the tool definitions near the start of the prompt. Everything after that point stops matching the cache Does dynamic tool discovery break prompt caching benefits?. The proposed fix is to give the agent a small, fixed set of general-purpose functions and let it write code for anything else. That keeps the start of the prompt the same while still allowing new capabilities. In other words, adding tools isn't the problem; changing the tool list partway through a session is. A large tool list that never changes caches fine.
At larger scale, the case for caching gets stronger. In a 115-day study of a long-running agent, 82.9% of all tokens were read from the cache Do persistent agents really cost less per token?. Once context stays put that consistently, the authors argue that cost per finished piece of work is a better measure than cost per token. The same idea of keeping repeated text shared, rather than reprocessing it, shows up at the server level too. PagedAttention stores the model's working memory in blocks that different requests can share Can OS paging techniques reduce LLM serving memory waste?.
There is also a different way to make prompts cheaper: shrink them. LLMLingua compresses prompts up to 20 times with little loss in accuracy Can prompts be compressed without losing model performance?, and ReWOO-style designs keep the prompt from growing after every tool call Can reasoning and tool execution be truly decoupled?. Be careful combining these with caching. Compression that rewrites the start of the prompt differently on each call can destroy the exact match that caching depends on. So when you see a cost figure, ask whether it came from making the prompt shorter or from keeping it identical, because the two approaches can work against each other.
Sources 6 notes
A benchmark across OpenAI, Anthropic, and Google found caching reduces API costs 41–80% and improves time-to-first-token 13–31% in multi-turn agentic tasks. However, naive full-context caching can increase latency by caching dynamic content that will never be reused.
MCP-style dynamic tool discovery breaks prompt caching because tool definitions in the prefix must match exactly for cache hits. Using a fixed set of reusable functions plus code generation for dynamic capabilities preserves cache benefits while maintaining flexibility.
A 115-day case study found 82.9% of tokens were cache reads. When context persists and reuses, the meaningful cost denominator becomes completed artifacts, not individual tokens.
PagedAttention applies OS virtual memory paging to the KV cache, storing blocks non-contiguously and enabling sharing across requests. This eliminates ~30% of memory waste from fragmentation, lifting vLLM throughput 2-4× over FasterTransformer and Orca on the same latency.
LLMLingua achieves up to 20x prompt compression with only 1.5-point performance drops by allocating less compression to instructions and questions, more to demonstrations, then pruning tokens using a small model's perplexity scoring aligned to the target LLM.
Show all 6 sources
ReWOO and Chain-of-Abstraction both decouple reasoning from tool responses through different mechanisms—planning-before-execution and abstract placeholders respectively—eliminating quadratic prompt growth and sequential latency while maintaining reasoning quality.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Don't Break the Cache: An Evaluation of Prompt Caching for Long-Horizon Agentic Tasks
- LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference
- Persistent AI Agents in Academic Research: A Single-Investigator Implementation Case Study
- Toward Efficient Agents: A Survey of Memory, Tool Learning, and Planning
- Towards a Science of Scaling Agent Systems
- Hogwild! Inference: Parallel LLM Generation via Concurrent Attention
- LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models
- Efficient Memory Management for Large Language Model Serving with PagedAttention