Can you shrink what an AI has to 'read' by just rewriting its instructions, instead of expensively retraining it?
How does prompt-based compression compare to reinforcement learning approaches in cost and effectiveness?
This explores whether you can shrink what an LLM has to read (prompts, agent histories, memory) more cheaply by rewriting instructions than by retraining the model with reinforcement learning, and which approach actually works better.
This explores whether compressing an LLM's context through clever prompting holds up against training-heavy reinforcement learning approaches, both on price and on results. The short version: the corpus has strong evidence that prompt-side methods are remarkably cheap and effective, but it has no clean head-to-head study pitting them against an RL-trained compressor on the same task. So treat what follows as a map of the trade-offs, not a final verdict.
The cheapest end is striking. Microsoft's ACON doesn't touch model weights at all. It reads cases where the agent failed, works out what the compression step threw away that it shouldn't have, and rewrites the plain-English compression guidelines to match. That cuts the agent's peak token usage by 26–54% and improves task success, for under $2 per benchmark, and it works on closed proprietary models you could never fine-tune anyway Can prompt optimization cut agent token usage without retraining?. One step earlier in the pipeline, LLMLingua compresses prompts up to 20x with only about a 1.5-point accuracy drop. The trick is to protect instructions and questions and squeeze the worked examples hardest, with a small model scoring which tokens matter Can prompts be compressed without losing model performance?. If your goal is purely lower API bills, prompt caching goes further still: it cuts costs 41–80% without dropping any content, though caching parts of the context that change every turn can actually slow things down Does prompt caching actually reduce agentic LLM costs?.
Training-based approaches buy something different: compression the model does on its own and adapts as it goes. DeepAgent learns to fold its own interaction history into structured episodic, working, and tool memories, and it can pause to rethink its strategy partway through a task Can agents compress their own memory without losing critical details?. SkillRL learns that successes and failures deserve different handling. It keeps successful runs as concrete examples and boils failures down to short lessons, which reaches top performance with much less context than treating every episode the same way Should successful and failed episodes be processed differently?. Notice the overlap, though: ACON also learns from failures. It just stores those lessons as editable text instead of in model weights. The two families are closer in spirit than their price tags suggest.
RL also carries costs that never appear in a compute bill. When the reward barely varies across attempts at a prompt, the learning signal weakens and the model can collapse into generic, one-size-fits-all outputs. Fixing that takes extra machinery, such as filtering for high-variance prompts Why do language models collapse into generic templates?. RL also needs a reward that measures compression quality, which is hard when what counts as 'important' is subjective. Splitting the judgment into checklists of specific criteria helps Can breaking down instructions into checklists improve AI reward signals?, but that is another piece you have to build. With prompt optimization, the guideline itself acts as the reward specification, and you can read it.
The finding you might not expect: how hard to compress depends less on which method you pick and more on the environment. ACON found that aggressive pruning works best in noisy, wordy settings, while tasks where one small fact decides the outcome need a conservative touch, whatever the agent's strength Should compression be aggressive or conservative in different environments?. The same idea appears in work on spending inference compute according to how hard each prompt is Can we allocate inference compute based on prompt difficulty?. A practical rule of thumb from the corpus is to start with prompt-level compression and caching, which cost almost nothing and can be inspected. Reach for RL when you need compression that adapts across long, open-ended agent runs and you can afford to design a reward that won't collapse.
Sources 9 notes
Microsoft's ACON method reduces agent token consumption and improves task success by iteratively refining compression guidelines through failure analysis, not weight updates. This approach works across proprietary LLM agents and costs under $2 per benchmark.
LLMLingua achieves up to 20x prompt compression with only 1.5-point performance drops by allocating less compression to instructions and questions, more to demonstrations, then pruning tokens using a small model's perplexity scoring aligned to the target LLM.
A benchmark across OpenAI, Anthropic, and Google found caching reduces API costs 41–80% and improves time-to-first-token 13–31% in multi-turn agentic tasks. However, naive full-context caching can increase latency by caching dynamic content that will never be reused.
DeepAgent's autonomous memory folding consolidates interaction history into episodic, working, and tool memory schemas. This reduces token overhead while letting agents pause to reconsider strategies—the autonomy and structure together avoid degradation that plagues poorly designed consolidation.
SkillRL demonstrates that treating successful episodes as concrete demonstrations and failures as abstracted lessons achieves state-of-the-art performance on complex tasks while using substantially less context than uniform approaches. The asymmetry mirrors human expert reasoning and avoids the degradation seen in uniform consolidation methods.
Show all 9 sources
When within-prompt reward variance is low, task gradients weaken and regularization dominates, pushing policies toward generic outputs. SNR-Aware Filtering—selecting high-variance prompts before updates—recovers performance across tasks and scales.
RLCF and RaR methods decompose instruction quality into verifiable sub-criteria, improving performance on benchmarks like FollowBench and HealthBench. This decomposition principle reduces overfitting to superficial artifacts that plague holistic reward models.
ACON's guideline optimization shows that aggressive pruning works best in noisy, verbose environments like AppWorld, while conservative retention matters more in high-fidelity tasks like OfficeBench where subtle facts drive decisions. The choice depends on the task's information structure, not the model's capability.
Research shows inference effectiveness varies dramatically by prompt difficulty. Reallocating the same total compute adaptively—giving easy prompts less and hard ones more—substantially outperforms larger models under uniform budgets.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- ACON: Optimizing Context Compression for Long-horizon LLM Agents
- LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models
- Don't Break the Cache: An Evaluation of Prompt Caching for Long-Horizon Agentic Tasks
- Useful Memories Become Faulty When Continuously Updated by LLMs
- Mechanistically Interpreting the Role of Sample Difficulty in RLVR for LLMs
- Inference-Aware Prompt Optimization for Aligning Black-Box Large Language Models
- RLVMR: Reinforcement Learning with Verifiable Meta-Reasoning Rewards for Robust Long-Horizon Agents
- Know It, Act on It: Investigating Memory Utilization in LLM Personalization