Can prompt optimization cut agent token usage without retraining?

Agent Context Optimization (ACON) compresses both the growing interaction history and incoming observations of a long-horizon LLM agent, but it does so by optimizing a natural-language "compression guideline" rather than updating any model's weights. The paper reports that on AppWorld, OfficeBench, and Multi-objective QA — three benchmarks each requiring "15+ interaction steps" — ACON "reduces peak token usage by 26–54% while improving task success over existing compression baselines." The authors frame the two bottlenecks this is meant to solve as "prohibitive inference memory costs" from the KV cache and "reasoning degradation due to irrelevant information," i.e., unbounded context is expensive to hold and dilutes the signal the agent needs to act on.

The mechanism is prompt optimization, not reinforcement learning on the agent or compressor. ACON "iteratively refines compression guidelines based on failure analysis of the agent," and the paper's ablation finds "the default o3 with contrastive feedback yields the best performance," while dropping contrastive feedback or swapping to a weaker optimizer model lowers accuracy — though a weaker optimizer like gpt-4.1 "still yields sufficiently fine guideline compared to the baseline guideline." Because the guideline lives in text, it never touches the underlying LLM's parameters, which the authors say is what makes ACON "purely model-agnostic and directly applicable to agents based on proprietary, API-based LLMs" where gradient updates aren't an option. The optimized guideline can then be distilled into a smaller compressor model, "preserving 95% of the teacher's accuracy," and the paper reports the optimization step itself costs "less than $2 per benchmark" — cheap next to the roughly "$20" of rollout cost for collecting AppWorld training data with gpt-4.1, and cheaper still than RL approaches like GRPO that need extensive rollouts for advantage estimation. With a smaller, cheaper compressor supplying concise context, the paper reports small models act as better long-horizon agents: "32% on AppWorld, 20% on OfficeBench, and 46% on Multi-objective QA" by "mitigating the distraction of long contexts."

This sits alongside Can an external manager handle context for frozen agents?, which also keeps the agent frozen and externalizes context control, but trains its manager with end-to-end reinforcement learning on a frozen agent rather than optimizing a prompt in language space. The two papers converge on separating compression from the agent model, by different routes: AdaCoM pays RL rollout cost to learn a manager policy; ACON pays a few dollars of prompt-optimization cost against failure cases to write a better guideline, then distills it into a cheap model. Where AdaCoM's compression aggressiveness tracks the frozen agent's own reliability, ACON's result points to a second axis — the environment's noise level — treated separately in the source's discussion of when to prune harder versus retain more.

The excerpt evaluates ACON "primarily... on GPT models due to resource constraints," so how the guideline-optimization approach transfers to other model families, or to agentic settings beyond these three benchmarks, is not established here. The paper also names its own remaining limitation plainly: "the compression process itself introduces computational overhead and increased latency," so the gains in token usage and task success are not free of added compute cost at inference time.