When an AI agent only hears 'pass' or 'fail' at the very end, why does learning cost so much more compute?
Why does outcome-only reinforcement learning need more than double the tokens to train agents?
This explores why training an agent on only a final success-or-failure reward costs so much more compute than methods that also reward or explain intermediate steps. The corpus has no note reporting the specific 'more than double' figure, so this covers the likely mechanisms, not the measurement.
This explores why an agent that only learns from a final win/lose signal burns so many more tokens than one that gets feedback along the way. No note here reports the 'more than double' number, but several notes point to the same causes. The nearest hard evidence is RLVMR, which adds programmatic rewards for planning, exploring, reflecting and monitoring. It cuts repetitive actions by 31% compared with outcome-only training. An agent that hears only 'you failed' at the end of a long episode has no reason to stop looping, so a large share of every rollout is redundant steps, and each step costs tokens. Can RL agents learn to reason better, not just succeed?
The deeper problem is how little a scalar reward carries. It is one number spread over hundreds of steps and thousands of tokens, and the model has to work out which step mattered by trying again and again. SDPO shows how much this discards. When the environment's actual feedback (the error message, the failed test) is fed back in context, the model can act as its own process reward model and produce dense, per-token credit. Outcome-only training rediscovers through many extra rollouts what one piece of feedback could have said directly. Can environment feedback replace scalar rewards in policy learning?
Many of those extra rollouts teach almost nothing. When rewards barely vary across attempts at the same task, the learning gradient weakens and regularization takes over. The fix is to filter for high-variance prompts before updating. That note is about reasoning models collapsing into generic templates, but the arithmetic carries over to agents. In a hard task with only an end reward, early episodes mostly fail the same way, so much of the sampling budget produces near-zero signal. Why do language models collapse into generic templates?
Outcome rewards also can't tell you where to spend effort. Across eight models, RL training moves from mastering execution to a phase where strategic planning is the bottleneck, and concentrating optimization on the planning tokens is what yields the gains. An outcome-only reward spreads credit evenly over every token, including execution steps the model already handles. You pay for the whole trajectory to improve a handful of decision points. Does RL training follow a predictable two-phase learning sequence?
The outcome signal itself isn't the expensive part. Reflexion shows a plain success/failure bit is enough when the agent writes a verbal diagnosis of what went wrong and keeps it in memory, with no gradient updates. SkillRL gets strong results with less context by keeping successes as concrete demonstrations and compressing failures into short lessons. The cost comes from using the outcome as a bare number that gradients must decode, when the agent could reason about it directly. Can agents learn from failure without updating their weights? Should successful and failed episodes be processed differently?
Sources 6 notes
RLVMR uses structured meta-reasoning tags (planning, exploration, reflection, monitoring) with programmatic rewards to train agentic RL. This reduces repetitive actions by 31% compared to outcome-only methods while maintaining better generalization than supervised fine-tuning alone.
SDPO converts tokenized environment feedback into dense gradient signals by using the feedback-conditioned policy as a self-teacher. The policy, when given retrospective evidence of its mistakes in-context, implicitly acts as its own process reward model, making external reward signals unnecessary.
When within-prompt reward variance is low, task gradients weaken and regularization dominates, pushing policies toward generic outputs. SNR-Aware Filtering—selecting high-variance prompts before updates—recovers performance across tasks and scales.
Across eight models, RL training consistently shows a first phase where execution correctness drives learning, followed by a second phase where strategic planning becomes the bottleneck. Planning token entropy increases while execution entropy stabilizes, and concentration of optimization on planning tokens yields significant performance gains.
Reflexion demonstrates that unambiguous environmental feedback (success/failure) enables agents to write useful self-diagnoses and improve across episodes without parameter updates. The binary signal prevents rationalization, and keeping reflections uncompressed preserves their usability.
Show all 6 sources
SkillRL demonstrates that treating successful episodes as concrete demonstrations and failures as abstracted lessons achieves state-of-the-art performance on complex tasks while using substantially less context than uniform approaches. The asymmetry mirrors human expert reasoning and avoids the degradation seen in uniform consolidation methods.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- RLVMR: Reinforcement Learning with Verifiable Meta-Reasoning Rewards for Robust Long-Horizon Agents
- AgentFly: Fine-tuning LLM Agents without Fine-tuning LLMs
- Useful Memories Become Faulty When Continuously Updated by LLMs
- RAGEN-2: Reasoning Collapse in Agentic RL
- From Trial-and-Error to Improvement: A Systematic Analysis of LLM Exploration Mechanisms in RLVR
- Mechanistically Interpreting the Role of Sample Difficulty in RLVR for LLMs
- Self-distillation Enables Continual Learning
- Efficient Reinforcement Learning via Large Language Model-based Search