Telling an AI only 'you failed' after 40 steps leaves it unable to figure out which step went wrong.
Why do outcome-only rewards fail to optimize long-horizon agent behavior?
This explores why a single reward at the end of a task — success or failure, nothing more — turns out to be a poor training signal for agents that act over many steps, and what the corpus offers instead.
This explores why a single pass/fail reward delivered at the end of a long task is a weak teacher for agents, and the collection converges on one root cause: an outcome-only reward is a scalar, and a scalar throws away most of what the agent needed to know. One note makes this precise by splitting agent feedback into two orthogonal parts — *evaluative* (how well did that go) and *directive* (what should change next time) Can scalar rewards capture all the information in agent feedback?. An outcome reward captures the first and discards the second. So the agent learns *that* a 40-step trajectory failed, but not *which* step broke it or how to fix it. That missing 'why' is exactly what makes numerical rewards stall: models stuck on a plateau start solving problems again the moment they're handed a chain-of-thought critique instead of a number Can natural language feedback overcome numerical reward plateaus?.
The long-horizon part sharpens the problem into a credit-assignment failure. When reward arrives only at the end, every one of the intermediate actions gets the same undifferentiated signal, and the agent can't tell the pivotal move from the filler. Several notes attack this by manufacturing a *dense* per-step signal the outcome reward never provided: tracking how much each turn shifts the agent's own belief toward the answer, which yields intrinsic credit without any critic network Can an agent's own beliefs guide credit assignment without critics?; or rewarding the metacognitive moves themselves — planning, exploration, reflection — which cuts repetitive floundering by 31% versus outcome-only training Can RL agents learn to reason better, not just succeed?. The through-line is that the reward has to reach *inside* the trajectory, not just sit at the end of it.
There's a subtler failure too: outcome-only rewards actively teach bad habits. Binary correctness rewards never punish a confident wrong answer, so they train the model to guess boldly and degrade its calibration — patchable only by adding a proper scoring term alongside the outcome Does binary reward training hurt model calibration?. And positive-only outcome reinforcement collapses diversity by piling probability mass onto whatever worked once, which quietly hurts an agent's ability to explore alternatives over a long horizon — one reason training on *negative* signals alone can match or beat full RL Does negative reinforcement alone outperform full reinforcement learning?.
What's striking is that the trait most predictive of long-horizon success isn't captured by outcome rewards at all. Across 17 frontier models, the winning behavior was *persistence* — staying productively in the benchmark-edit-incorporate loop rather than quitting or burning budget What predicts success in ultra-long-horizon agent tasks?. An end-of-task reward gives you no gradient toward 'keep iterating usefully'; it only tells you whether you eventually arrived. That's why the corpus keeps routing around scalar outcomes entirely — treating successes and failures asymmetrically as demonstrations vs. abstracted lessons Should successful and failed episodes be processed differently?, or moving credit assignment into episodic memory so an agent improves continually without ever updating weights Can agents learn continuously from experience without updating weights?.
The thing you might not have expected: the fix for outcome-only reward isn't a better number, it's usually a *different kind* of signal. Nearly every method here recovers something the scalar deleted — the directive 'how,' the per-step 'when,' the confidence 'how sure,' the persistence 'keep going.' Outcome rewards don't fail because they're wrong about the outcome; they fail because a long task is a sequence of decisions, and one bit at the end can't grade a sequence.
Sources 9 notes
Natural feedback carries two orthogonal types of information: evaluative (how well an action performed) and directive (how it should change). Scalar rewards capture evaluation but discard directional specifics that token-level distillation can recover, making the two complementary rather than redundant.
Critique-GRPO shows that models stuck on performance plateaus can generate correct solutions when given chain-of-thought critiques, revealing that numerical rewards lack critical information about why failures occur and how to improve.
ΔBelief-RL uses log-ratios of sequential probability estimates to assign per-turn credit without critic networks or process reward models. Tested on 20 Questions, smaller models trained this way matched or exceeded prior SOTA and larger baselines while generalizing beyond training.
RLVMR uses structured meta-reasoning tags (planning, exploration, reflection, monitoring) with programmatic rewards to train agentic RL. This reduces repetitive actions by 31% compared to outcome-only methods while maintaining better generalization than supervised fine-tuning alone.
Binary correctness rewards incentivize high-confidence guessing because they don't penalize confident wrong answers. Adding the Brier score as a second reward term mathematically guarantees joint optimization of accuracy and calibration without trade-off.
Show all 9 sources
Training with only negative samples consistently improves Pass@k across the spectrum, often matching full PPO and GRPO. Negative reinforcement suppresses incorrect trajectories while preserving diversity, whereas positive-only reinforcement degrades higher-k performance by concentrating probability mass.
Across 17 frontier models on 36 expert-curated optimization tasks, repeated benchmark-edit-incorporate cycles within a wall-clock budget proved the dominant success predictor. Most models terminated early or burned budget unproductively; Claude Opus 4.6 stood out as persistent.
SkillRL demonstrates that treating successful episodes as concrete demonstrations and failures as abstracted lessons achieves state-of-the-art performance on complex tasks while using substantially less context than uniform approaches. The asymmetry mirrors human expert reasoning and avoids the degradation seen in uniform consolidation methods.
AgentFly formalizes agent learning as a Memory-augmented MDP with three memory modules (case, subtask, tool) that enable credit assignment and policy improvement entirely through memory operations. The approach achieved 87.88% on GAIA validation without modifying LLM parameters.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Reward Reasoning Model
- RLVMR: Reinforcement Learning with Verifiable Meta-Reasoning Rewards for Robust Long-Horizon Agents
- AgentFly: Fine-tuning LLM Agents without Fine-tuning LLMs
- ReasoningBank: Scaling Agent Self-Evolving with Reasoning Memory
- Useful Memories Become Faulty When Continuously Updated by LLMs
- SkillRL: Evolving Agents via Recursive Skill-Augmented Reinforcement Learning
- The Landscape of Agentic Reinforcement Learning for LLMs: A Survey
- Learning to Reason without External Rewards