Don't Break the Cache: An Evaluation of Prompt Caching for Long-Horizon Agentic Tasks

Recent advancements in Large Language Model (LLM) agents have enabled complex multi-turn agentic tasks requiring extensive tool calling, where conversations can span dozens of API calls with increasingly large context windows. However, although major LLM providers offer prompt caching to reduce cost and latency, its benefits for agentic workloads remain underexplored in the research literature. To our knowledge, no prior work quantifies these cost savings or compares caching strategies for multi-turn agentic tasks. We present a comprehensive evaluation of prompt caching across three major LLM providers (OpenAI, Anthropic, and Google) and compare three caching strategies, including full context caching, system prompt only caching, and caching that excludes dynamic tool results. We evaluate on DeepResearch Bench, a multi-turn agentic benchmark where agents autonomously execute realworld web search tool calls to answer complex research questions, measuring both API cost and time to first token (TTFT) across over 500 agent sessions with 10,000-token system prompts. Our results demonstrate that prompt caching reduces API costs by 41-80% and improves time to first token by 13-31% across providers. We find that strategic prompt cache block control, such as placing dynamic content at the end of the system prompt, avoiding dynamic traditional function calling, and excluding dynamic tool results, provides more consistent benefits than naive fullcontext caching, which can paradoxically increase latency. An ablation study across prompt sizes (500-50,000 tokens) and tool call counts (3-50) demonstrates universal linear cost and TTFT benefits, after the provider caching token minimum, and reveal provider-specific strategy discrepancies across variants. We provide nuanced discussion and guidance for implementing prompt caching in production agentic systems.

Introduction. Recent advancements in Large Language Model (LLM) agents have enabled complex, long-horizon agentic tasks that require extensive tool calling across multi-turn conversations (Ji, 2025). Through function calling, LLM agents can invoke APIs, execute web searches, interact with databases, and perform domain-specific actions on behalf of users. As these agentic workloads grow in complexity, conversations can span dozens of API calls with context windows accumulating tens of thousands of tokens, leading to significant costs and latency overhead. To address this, major LLM providers including OpenAI, Anthropic, and Google offer prompt caching, a feature that reuses previously computed key-value (KV) tensors from attention layers to avoid redundant computation on repeated prompt prefixes (OpenAI, 2026; Anthropic, 2026; Google Cloud, 2026b).

While providers offer reduced pricing for cached input tokens, the benefits of prompt caching in real-world agentic workloads remain under-explored in the research literature. Existing work on KV cache optimization focuses primarily on inference-level memory management and compression (Kwon et al., 2023; Ge et al., 2023; Shi et al., 2024), rather than evaluating the enterprise-grade prompt caching features offered through provider APIs. Concurrent work has audited prompt caching across providers to detect timing side-channel vulnerabilities (Gu et al., 2025), but to our knowledge, no prior work has quantified the cost benefits of prompt caching or compared caching strategies for agentic workloads. This gap is particularly significant given the recent proliferation of long-running agents for deep research, coding assistance, and autonomous task completion, where prompt caching could substantially reduce operational costs and improve user experience through faster response times.

In this paper, we present the first comprehensive evaluation of prompt caching strategies for long-horizon agentic tasks across three LLM providers (OpenAI, Anthropic, and Google) using four flagship models (Figure 1). We compare three caching strategies, including full context caching, system prompt only caching, and caching that excludes dynamic tool results. We evaluate on DeepResearch Bench (Du et al., 2025), a multi-turn agentic benchmark where agents autonomously execute web search tool calls to answer research questions. Our evaluation spans 500 agent sessions with 10,000-token system prompts, measuring both API cost and time to first token (TTFT) across all conditions.

Our evaluation reveals three key findings:

Prompt caching delivers substantial and consistent cost savings across all providers: All four models tested show statistically significant cost reductions when prompt caching is enabled. Cost savings range from 41% to 80% across providers. These savings are consistent across all three caching strategies, demonstrating that prompt caching provides reliable cost benefits regardless of the specific caching approach employed.

Latency improvements vary significantly across providers and require careful strategy selection: Time to first token improvements range from 13% to 31% across providers, though latency variance differs substantially between providers. Notably, the cache strategy that maximizes cost savings does not always maximize latency improvement, highlighting the importance of strategy selection based on optimization goals.

Strategic cache boundary control outperforms naive fullcontext caching: Providers abstract much of the caching mechanism, automatically triggering cache creation when token thresholds are exceeded. However, naively enabling full-context caching can paradoxically increase latency, as dynamic tool calls and results may trigger cache writes for content that will not be reused across sessions. By strategically controlling cache boundaries, such as caching only the system prompt or explicitly excluding tool results, practitioners can ensure that only stable, reusable content is cached. Our results show that system prompt only caching provides the most consistent benefits across both cost and latency dimensions.

Method. Recent advances in LLM agents have enabled complex, long-horizon tasks that extend far beyond single-turn question answering. Modern agentic applications including deep research assistants, coding agents such as Claude Code and Cursor, and autonomous task completion systems like Manus routinely execute 30-50 or more tool calls within a single session (Ji, 2025; Du et al., 2025; Mialon et al., 2023; Zhou et al., 2023; Drouin et al., 2024; Wei et al., 2025). In such workflows, each tool call adds content to the conversation context, including the tool invocation, execution results, and the model’s subsequent reasoning, causing context windows to grow rapidly throughout the session.

This growth presents challenges for prompt caching (Guan et al., 2026; Laban et al., 2025). Recent work has proposed solutions for multi-turn caching scenarios (Jeong & Ahn, 2025; Yan et al., 2025). Unlike static question-answering scenarios where prompts are largely predetermined, agentic workloads feature dynamic, session-specific content that accumulates unpredictably. Tool results often contain userspecific data that will not benefit other sessions, and the interleaving of static system prompts with dynamic tool outputs complicates cache reuse. Context engineering strategies have emerged to manage these challenges, including treating external storage as extended memory and structuring prompts to maximize cache efficiency (Ji, 2025; Lumer et al., 2025a). However, the effectiveness of prompt caching across different caching strategies in agentic workloads has not been comprehensively evaluated. Our work addresses this gap by measuring cost and latency benefits across controlled caching strategies on a multi-turn agentic benchmark.

3. Methodology 3.1. Experimental Setup We evaluate prompt caching across three major LLM providers: OpenAI, Anthropic, and Google. For each provider, we select flagship models: GPT-5.2 from OpenAI, Claude Sonnet 4.5 from Anthropic, and Gemini 2.5 Pro from Google. We additionally include GPT-4o to examine caching behavior across model generations. All selected models support prompt caching through their respective APIs. use a 10,000-token system prompt containing agent instructions for deep research, including guidance on tool usage, question decomposition, and report synthesis. Each session starts with a fresh context, ensuring that cache benefits are measured within individual multi-turn conversations rather than across sessions.

3.2. Cache Mode Implementation We implement four cache conditions to systematically evaluate prompt caching strategies (see Appendix C, Figures 6–9 for visual illustrations). To control cache boundaries precisely, we use unique identifiers (UUIDs) to break the cache at specific points in the prompt, ensuring that content after the UUID is not cached from previous requests.

No Cache (Baseline): A UUID is prepended to the beginning of the system prompt, breaking the cache immediately and forcing the model to recompute all tokens. This serves as our baseline condition where no caching benefits are realized. In real-world agentic tasks, this symbolizes including dynamic content, such as timestamps and user information, to the system prompt on inference time (Ji, 2025).

Full Context Caching: No UUIDs are added, allowing the provider’s caching mechanism to operate automatically. OpenAI and Google enable prompt caching automatically for eligible requests, while Anthropic requires explicit cache breakpoints in the API request. This condition represents naive caching where practitioners enable the feature without additional optimization.

System Prompt Only Caching: A UUID is appended to the end of the system prompt, breaking the cache at this boundary. This ensures that only the static system prompt is cached, while the dynamic conversation history, tool calls, and tool results are recomputed on each request.

Exclude Tool Results Caching: UUIDs are appended both after the system prompt and after each tool result. This strategy ensures that tool results, which are dynamic and session-specific, do not contribute to the cache.

Discussion. 5.1. Strategic Cache Boundary Control Our results demonstrate that strategic control over cache boundaries is essential for maximizing prompt caching benefits in agentic workloads. The key insight is that providers abstract much of the underlying caching mechanism, automatically triggering cache creation when token thresholds are exceeded. Without explicit boundary control, this automatic behavior can cache dynamic, session-specific content that will not be reused, leading to cache write overhead without corresponding read benefits.

The most effective strategy is to ensure that only stable, reusable content is cached. In agentic applications, the system prompt is the most stable component, containing agent instructions, tool definitions, and persona guidelines that remain constant across sessions. Conversation history, tool calls, and tool results are dynamic and session-specific, making them poor candidates for cross-session caching. Practitioners should avoid including dynamic values in the system prompt itself. Common patterns that inadvertently break the cache include timestamps, datetime strings, session identifiers, or user-specific information embedded in the system prompt. If such dynamic information is necessary, it should be placed at the end of the system prompt to maximize the cacheable prefix. This ensures that the majority of the system prompt benefits from cache hits while only the dynamic suffix requires recomputation (Ji, 2025).

Similarly, dynamic function calling can break the cache when tool definitions change between requests. Modern agentic systems increasingly leverage dynamic tool discovery through protocols such as the Model Context Protocol (MCP) (Model Context Protocol, 2026), where available tools may vary based on connected servers or runtime context (Lumer et al., 2025a). When tool definitions are included in the prompt, any change to the available tool set invalidates the cached prefix. A practical strategy is to maintain a fixed set of general-purpose, reusable functions while implementing dynamic capabilities through code generation rather than traditional function calling (Ji, 2025; Wang et al., 2024). Prior methods of dynamic function calling can prevent prompt caching usage (Lumer et al., 2024; 2025b).

5.2. Tool Call Caching Considerations For long-running agentic sessions with 30-50+ tool calls, practitioners may consider caching tool calls and results to reduce costs. However, cache creation incurs overhead on the first request, which is only amortized if subsequent requests benefit from cache reads. For tool calls that produce variable results or are unlikely to be repeated, caching provides no benefit and may introduce overhead.

Common context management strategies can interact poorly with tool call caching. Techniques such as summarizing or pruning old tool calls (Ji, 2025) break cached representations, making tool call caching counterproductive. The emerging pattern is to maintain a stable system prompt that benefits from caching while treating tool calls as dynamic content that may be managed throughout the session.

5.3. Provider Implementation Variability Provider implementations of prompt caching differ in important ways that affect practical deployment. Minimum token thresholds for cache eligibility range from 1,024 to 4,096 tokens depending on the provider and model (see Appendix A). Time-to-live (TTL) durations vary from 5 minutes to 24 hours, affecting whether cached content remains available across user sessions. Some providers offer automatic caching that activates without developer intervention, while others require explicit API parameters to enable caching.

Conclusion. We present the first comprehensive evaluation of prompt caching for long-horizon agentic tasks across three major LLM providers. Our results demonstrate that prompt caching reduces API costs by 41 to 80% and improves TTFT by 13 to 31%. Strategic cache boundary control, such as caching only system prompts while excluding dynamic tool results, provides more consistent benefits than naive full context caching, which can paradoxically increase latency. These findings provide actionable guidance for deploying prompt caching in production agentic systems.