Why does letting an AI agent grab a new tool mid-task suddenly make the whole conversation slower and pricier?
Why does dynamic tool discovery break prompt cache reuse in agentic systems?
This explores why letting an agent find and load new tools in the middle of a task (as MCP-style setups do) stops the model provider from reusing the cached, already-processed start of the prompt, and what that costs.
This explores why letting an agent pick up new tools mid-task wipes out the savings from prompt caching, and what builders do about it. The mechanism is simple. Prompt caching only works when the start of the prompt is exactly the same, token for token, from one call to the next. Tool definitions usually sit in that opening section. When an agent discovers a new tool through MCP and adds its schema, the opening changes, the cache misses, and the whole context has to be processed again at full price Does dynamic tool discovery break prompt caching benefits?. The fix proposed there is to keep a small, fixed set of general functions and let the model write code when it needs a new capability. The flexibility moves out of the tool list and into what the agent generates.
This matters because caching is not a minor optimization for long-running agents. It is most of the bill. A 115-day case study of a persistent agent found that 82.9% of all tokens were cache reads Do persistent agents really cost less per token?. If a system's costs depend that heavily on a stable, reused context, anything that keeps changing the prompt prefix hits the main cost lever, not a side expense.
Dynamic discovery isn't just a bad habit, though. It solves a real problem. DeepAgent shows that agents finding tools as they go, instead of being handed a pre-chosen set, keep a better view of the whole task and adapt better on long tasks where there are too many tools to list up front Can agents discover tools dynamically instead of pre-selecting them?. So there is a genuine trade-off: being able to adapt at runtime pulls against the economics of a fixed prompt. Production experience leans toward stability for a separate reason too. Teams that dropped MCP for direct function calls, one tool per agent, did it because ambiguous tool selection made results unpredictable, not because of cost Why do protocol-based tool integrations fail in production workflows?. A tool list that keeps changing is harder to cache and harder to predict.
The less obvious lesson comes from the memory side of the corpus: you can add capabilities without changing the prompt prefix. VOYAGER stores skills as executable code in a searchable library that sits outside the prompt and builds complex skills from simpler ones Can agents learn new skills without forgetting old ones?. Agent Workflow Memory similarly extracts reusable sub-routines from past runs Can agents learn reusable sub-task routines from past experience?. Both point to the same design: keep the tool interface fixed, let the set of capabilities grow as code or data behind it, and load what's needed later in the context, not at the top. The cache-friendly agent and the lifelong-learning agent turn out to be the same architecture.
Sources 6 notes
MCP-style dynamic tool discovery breaks prompt caching because tool definitions in the prefix must match exactly for cache hits. Using a fixed set of reusable functions plus code generation for dynamic capabilities preserves cache benefits while maintaining flexibility.
A 115-day case study found 82.9% of tokens were cache reads. When context persists and reuses, the meaningful cost denominator becomes completed artifacts, not individual tokens.
DeepAgent demonstrates that discovering tools as needed—rather than pre-retrieving a fixed set—enables agents to maintain global task perspective and adapt strategy mid-execution. This approach scales better for long-horizon tasks where the tool space is too large to enumerate.
MCP integration caused non-deterministic failures through ambiguous tool selection and parameter inference. Replacing it with explicit direct function calls and single-tool-per-agent design restored determinism. A 306-practitioner survey confirms 85% of production teams build custom agents, forgoing frameworks.
VOYAGER demonstrates that storing executable skills in an embedding-indexed library and composing complex skills from simpler ones allows agents to learn continuously while avoiding the forgetting that occurs with weight-update-based methods. Environmental feedback refines skills while an automatic curriculum drives continual exploration.
Show all 6 sources
Agent Workflow Memory induces sub-task routines at finer granularity than full tasks, abstracts example-specific values, and compounds them hierarchically. This produces 24.6% relative gain on Mind2Web and 51.1% on WebArena, with larger gains as train-test gaps widen.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Demystifying Agent Skills: Why They Work-Until They Don't
- LiveMCP-101: Stress Testing and Diagnosing MCP-enabled Agents on Challenging Queries
- Don't Break the Cache: An Evaluation of Prompt Caching for Long-Horizon Agentic Tasks
- MUSE-Autoskill: Self-Evolving Agents via Skill Creation, Memory, Management, and Evaluation
- Why Do Multi-agent LLM Systems Fail?
- A Practical Guide for Designing, Developing, and Deploying Production-Grade Agentic AI Workflows
- Towards a Science of Scaling Agent Systems
- Agent Workflow Memory