ACON: Optimizing Context Compression for Long-horizon LLM Agents

Large language models (LLMs) are increasingly deployed as agents in dynamic real-world environments, where success depends on maintaining precise records of actions and observations. However, the resulting unbounded context growth in long-horizon agentic tasks makes two critical bottlenecks: prohibitive inference memory costs and reasoning degradation due to irrelevant information. Existing compression methods fail to fully address this, often relying on brittle heuristics or requiring parameter updates impractical for proprietary or large-scale LLMs. We introduce Agent Context Optimization (ACON), a unified framework that optimally compresses both observations and history into concise, informative representations. Distinct from prior works, ACON employs an optimization in natural language space: it iteratively refines compression guidelines based on failure analysis of the agent, ensuring critical state information is preserved without model fine-tuning. To further minimize computational overhead, we distill the optimized compressor into smaller models. Experiments on AppWorld, OfficeBench, and Multi-objective QA demonstrate that ACON reduces peak token usage by 26–54% while improving task success over existing compression baselines. Notably, it enables smaller LMs to function effectively as long-horizon agents, achieving up to 46% performance improvement by mitigating context distraction. Our code is available at https://github.com/microsoft/acon.

Introduction. Large language models (LLMs) have become the backbone of AI agents, enabling them to plan and act in dynamic environments (Yao et al., 2023). However, these tasks often unfold over extended horizons, requiring the agent to maintain a continuous record of observations, tool outputs, and evolving states. In such settings, context is not auxiliary but foundational; losing a single detail such as a file path or an API parameter can derail the entire workflow. As interactions accumulate, context grows unbounded as shown in Figure 2, making two major bottlenecks. First, the inference memory cost of transformers scales with context length, resulting long-horizon reasoning computationally prohibitive due to the massive KV cache requirements (Vaswani et al., 2017). Second, excessively long contexts dilute relevant information, distracting the model with outdated or extraneous details and degrading decision quality (Shi et al., 2023).

The challenge of managing this context is particularly critical in productivity scenarios, such as email management or workflow automation, where agents must coordinate across heterogeneous tools (Trivedi et al., 2024; Wang et al., 2024b). Unlike simpler conversational tasks, these environments demand the preservation of diverse signal types: factual history, action-outcome relationships, success preconditions, and future decision cues. Naive strategies like token truncation or generic summarization are insufficient, as they easily discard these critical details essential for multi- RQ1: How to optimize the compressor?

RQ2: How to reduce cost of compressor → Can we use small LM instead of LLM? step reasoning. Consequently, effective compression must balance aggressive reduction with the precise retention of task-relevant state information.

Existing compression approaches, however, fail to fully address these agent-specific needs. Dialogue-oriented systems (Packer et al., 2023) focus on conversational coherence rather than state tracking, while document-centric methods (Jiang et al., 2024) assume single-step reasoning where context can be discarded after use. While recent agent-focused methods attempt to bridge this gap, they face significant limitations: heuristic-based approaches (Deng et al., 2023; Smith, 2025) are often brittle and narrowly specialized, limiting their robustness. Meanwhile, model optimization-based approaches (Zhou et al., 2025; Sun et al., 2025) typically entangle compression with the agent model, making them difficult to apply directly to proprietary, API-based LLMs, where gradient-based updates to the underlying model are infeasible.

To address these challenges, we introduce Agent Context Optimization (ACON), a unified framework for optimizing the compression of both environment observations and interaction histories. Distinct from heuristic approaches that rely on static rules and model optimization-based methods that require model parameter updates, ACON introduces a compression guideline optimization directly in natural language space. This method refines compressor prompts through failure analysis ensuring that critical environmentspecific signals are preserved after compression without altering the agent model weight (Pryzant et al., 2023; Yang et al., 2024a; Yüksekgönül et al., 2025; Han et al., 2025). It makes ACON purely model-agnostic and directly applicable to agents based on proprietary, API-based LLMs.

ACON yields three key advantages over previous works. First, the guideline optimization enables environmentspecific compression rules to be derived consistently across diverse agentic tasks, overcoming the brittleness of handcrafted heuristics. Second, by retaining essential information, optimally compressed contexts not only reduce memory costs but also improve decision quality, allowing smaller models to act more effectively by mitigating distraction. Third, we validate that these optimized compressors can be distilled into smaller models, demonstrating that the compression module itself can be deployed with minimal computational overhead.

We validate ACON on three multi-step agent benchmarks: AppWorld (Trivedi et al., 2024), OfficeBench (Wang et al., 2024b), and Multi-objective QA (Kwiatkowski et al., 2019; Zhou et al., 2025), each requiring 15+ interaction steps. Our empirical results demonstrate clear advantages of ACON: (1) lowers peak token usage of agents by 26–54% while improving task success compared to existing compression baselines (2) enables effective distillation of the context compressor into smaller models, preserving 95% of the teacher’s accuracy, thereby reducing the overhead of the compression (3) allows small LMs to function more effectively as agents, improving performance by 32% on AppWorld, 20% on OfficeBench, and 46% on Multi-objective QA by mitigating the distraction of long contexts. Our result highlights on AppWorld benchmark are in Figure 1.

In summary, our work makes the following contributions:

Related work. Long-horizon LLM agents. Large language model (LLM) agents extend pretrained models beyond static single-step reasoning tasks (e.g., RAG-based QA, math problem solving, or code generation) to interactive decision-making in dynamic environments (Yao et al., 2023; Wang et al., 2024a; Team et al., 2025; OpenAI, 2025a). Unlike chatbots or solvers that return an answer in one pass, agents must iteratively observe their surroundings, select tools, and execute actions while revising their plans based on feedback (Shridhar et al., 2021; Jimenez et al., 2024; Zhou et al., 2024; Wei et al., 2025; Xie et al., 2024; Bonatti et al., 2024). Recent work highlights the importance of long-horizon LLM agents, which tackle tasks that unfold over dozens to hundreds of steps and require coordination across multiple applications and tools (Kwa et al., 2025; Trivedi et al., 2024; Wang et al., 2024b; 2025b). A central challenge in these scenarios lies in managing the dynamic long context, where the agent must retain multi-step interaction histories and handle diverse observations produced by heterogeneous environments.

Context compression for LLMs. Managing this evergrowing context has been a longstanding challenge, and a variety of approaches have been proposed to compress LLM inputs. Prior works on context compression can be broadly grouped into three directions: document- or retrieval-based compression (Seo et al., 2025; Li et al., 2023; Xu et al., 2024; Yoon et al., 2024; Zhou et al., 2025; Jiang et al., 2024; Shandilya et al., 2025), dialogue memory summarization (Xu et al., 2025; Maharana et al., 2024; Wang et al., 2025a), and low-level KV cache compression (Zhang et al., 2025). While each line of research has demonstrated benefits in its respective setting, they remain insufficient for the dynamic and heterogeneous contexts required by long-horizon agents, where the relevance of information frequently shifts as the agent progresses.

Beyond general compression, several recent works have explored context compression specifically for LLM agents (Deng et al., 2023; Lee et al., 2025; Smith, 2025; Yu et al., 2025). However, these approaches either rely on naive prompting or target narrow domains, limiting their broader applicability. Another related line of works treats context compression as an agent action (Zhou et al., 2025; Ye et al., 2025; Sun et al., 2025), employing reinforcement learning to optimize the model for both compression and agent action policy. However, such methods inherently update the model to couple reasoning with compression, typically requiring access to internal weights.

Method. 3. Agent Context Optimization (ACON) We present Agent Context Optimization (ACON), a unified framework for optimized history and observation compression in long-horizon LLM agents. We begin by formulating the agent task and defining context cost in Section 3.1. Next, in Section 3.2, we introduce generative compression with LLMs for both history and observation, and formalize the associated optimization objective and its challenges. We then propose our optimization method in Section 3.3, followed by a distillation that enables smaller models for compressions to reduce the compression cost (Section 3.4).

3.1. Problem Formulation Task. An agentic task is formulated as a Partially Observable Markov Decision Process (POMDP) E = ⟨S, A, O, T , R⟩with state space S, action space A, and observation space O. The transition function T (s, a) → (s′, o) is deterministic, and it is determined by the environment. Specifically, it executes an action a ∈A in the environment and returns the next state and observation. The reward function R returns the reward given the terminal state sT . The terminal state is arrived when the transition function receives the special action (e.g., finish_task).

An LLM agent interacts with the environment to get information for making a decision to achieve a given task ACON: Optimizing Context Compression for Long-horizon LLM Agents where θ refers to the pre-trained parameters of the LLM and Pagent is the prompt that consists of a general environment description, tools, output format, and few-shot examples in natural language.

3.2. History & Observation Compression with LLMs To address this challenge, we use an LLM f(·; φ, P), parameterized by pre-trained weights φ and a compression guideline P, to minimize context cost defined in Equation 2 (e.g., summarize the given interaction history). As in Equation 1, the LLM receives two inputs at each step: the interaction history ht−1 and the latest observation ot. This introduces two options for context compression:

History compression. The interaction history accumulates both environment observations and agent actions. In long-horizon tasks, this history can grow excessively large. To manage its length, we apply history compression only when the length exceeds a predefined threshold Thist: overhead of invoking the compressor is incurred only when necessary (Smith, 2025).

In both cases, the compressor LLM selects information to preserve based on its learned prior knowledge of importance. However, there is no guarantee that the salient details required for successful task completion are retained. The agent context effectively serves as a world model of the environment, encompassing diverse forms of information such as causal relations (e.g., email leaves drafts), evolving states (e.g., account balance), preconditions (e.g., login required), and task-relevant decision cues (e.g., due dates). Effective context compression must therefore accommodate this heterogeneous and dynamic nature of agent context, ensuring that the most critical signals are preserved for long-horizon reasoning and task success.

Challenges. The optimization objective in Equation 7 is difficult to optimize in practice because there is no gold supervision for compression, the reward is sparse and only revealed at the end of the trajectory, and the context cost is defined over discrete quantities, which precludes direct gradient computation. While these properties naturally motivate reinforcement learning (RL) (Sutton & Barto, 2018), applying RL introduces additional obstacles: (1) updating the parameters φ of a LLM with RL can be computationally prohibitive, (2) environment rollouts are extremely expensive since each reward requires multi-step executions of both agent and compressor, and (3) policy gradient estimates suffer from high variance since compression quality is only indirectly evaluated through eventual task success.

3.3. Optimizing Compression Guidelines To overcome these challenges, we propose to optimize compression guidelines P (natural language prompts) for context compression, rather than fine-tuning model parameters φ.

Discussion. For history compression, as shown in Table 1, on App- World, ACON reduces peak tokens by over 25% while preserving the accuracy of the no compression upper bound, outperforming all baselines that suffer severe degradation on medium and hard tasks spanning longer steps. On OfficeBench (Table 2a), ACON lowers peak context size by Applying only the utility maximization step (UT) improves performance while reducing token cost across all benchmarks, whereas the compression maximization step (CO) further lowers token cost but may slightly hurt accuracy depending on the environment. This trade-off suggests a practical guideline for choosing between the two variants. In verbose and noisy environments such as App- World, where observations contain redundant API outputs and distractors, UTCO is often preferable because aggressive pruning can improve both efficiency and focus. In contrast, in high-fidelity information-seeking tasks such as OfficeBench and 8-objective QA, UT is generally safer because over-compression may remove subtle facts needed for final decisions. Therefore, UTCO is recommended when observations contain substantial redundancy, while UT is preferable when precise fact retention is critical.

Prompt optimizer: o3 + contrastive feedback works best. We analyze how the choice of optimizer and the use of contrastive feedback affect compression guideline quality. As shown in Table 3, the default o3 with contrastive feedback yields the best performance, while removing contrastive feedback (only using failed trajectories) or switching to other models results in lower accuracy. Although o3 shows the best performance, we also demonstrate that the optimizer model can be replaced to weaker models such as gpt-4.1, showing it still yields sufficiently fine guideline compared to the baseline guideline.

Optimization cost: practical and lightweight. One might assume that leveraging a reasoning model like o3 would be cost-prohibitive. However, our guideline optimization step is remarkably lightweight, demanding less than $2 per benchmark. This can be negligible compared to the expense of trajectory rollouts on the training dataset. For instance, utilizing gpt-4.1 on AppWorld train set requires a total rollout cost of approximately $20. Furthermore, even this data collection cost is substantially lower than that of RL-based methods such as GRPO (Shao et al., 2024), which requires extensive rollout for advantage estimation. Detailed API cost computation and breakdowns are provided in Section B.2 and C.2.

Practical efficiency trade-off. We further quantify the practical overhead of generative compression on App- World. For API cost, we estimate the end-to-end cost of

Conclusion. We presented Agent Context Optimization (ACON), a unified framework that systematically compresses both interaction histories and environment observations for longhorizon LLM agents. Unlike prior work that relies on naive prompting or narrow domains, ACON introduces compression guideline optimization in natural language space, enabling adaptive and model-agnostic compression. Experiments on AppWorld, OfficeBench, and Multi-objective QA show that ACON reduces peak tokens by 26–54% while improving task success over existing compression baselines, with small degradation relative to full-context baseline. Beyond memory efficiency, we demonstrate that optimized compressors can be distilled into smaller models, substantially lowering overhead without sacrificing performance. Moreover, by supplying concise yet informative contexts, ACON allows small agents such as Qwen3-14B to approach the performance of much larger models. Overall, our findings highlight that ACON lays a foundation for more general, memory-efficient, and deployable longhorizon LLM agents.

Limitations. and Future Work. While ACON effectively reduces context costs, a few limitations remain. First, our empirical evaluation primarily focuses on GPT models due to resource constraints. Second, like many context management frameworks, the compression process itself introduces computational overhead and increased latency. We include more detailed discussion on limitations and future work in Appendix A.