Giving an AI more frequent training rewards doesn't help much — only about 20% of its reasoning tokens are real decision points anyway.
How does reward density during training affect token efficiency in reasoning?
This explores whether the richness of the training signal — sparse end-of-answer rewards versus dense per-step feedback — changes how economically a model spends its reasoning tokens, and the corpus suggests the surprising answer is that the signal is already concentrated in a tiny slice of tokens no matter how the reward is delivered.
This explores how reward density (one signal at the end of an answer versus feedback along the way) affects token efficiency (how many reasoning tokens a model burns to get there). The corpus reframes the question in a useful way: before you ask how dense the *reward* should be, notice that the *learning signal* inside a reasoning chain is already extremely sparse. Only about 20% of tokens are high-entropy 'forking points' where the model actually decides where to go, and RLVR overwhelmingly adjusts those — training on just that minority matches or beats full-gradient updates Do high-entropy tokens drive reasoning model improvements?. A parallel finding shows specific tokens like 'Wait' and 'Therefore' are mutual-information peaks that carry the signal about correct answers, while most surrounding tokens don't Do reflection tokens carry more information about correct answers?. So a single sparse outcome reward isn't spread thinly across a chain — it effectively lands on a handful of pivotal tokens.
That sparseness cuts against the assumption that longer chains mean better reasoning. Accuracy actually peaks and then declines as thinking tokens grow — one setup dropped from 87% to 70% as tokens climbed from ~1,100 to ~16K, because models overthink easy problems and underthink hard ones Does more thinking time always improve reasoning accuracy?. And when you ask which tokens are worth keeping, models rank them by function: symbolic computation is preserved, grammar and meta-discourse get pruned first, and students trained on the pruned chains outperform those trained on verbose frontier-model traces Which tokens in reasoning chains actually matter most?. Token efficiency, in other words, is less about how much reward you pour in and more about whether the reward routes to the tokens that do computational work.
Where reward density genuinely matters is in escaping plateaus, not in trimming tokens. Pure numerical rewards — the sparsest kind — carry no information about *why* an answer failed, and models get stuck. Adding chain-of-thought critiques (a denser, language-shaped signal) lets stalled models suddenly produce correct solutions Can natural language feedback overcome numerical reward plateaus?. Denser evaluation on the reward side helps too: process/step-level judges that reason about each step outperform classifier-style reward models and do it with far less training data Can judges that reason about reasoning outperform classifier rewards?, and reward models that reason before scoring raise their own accuracy ceiling Can reward models benefit from reasoning before scoring?.
The twist that should stick with you is how little the *content* of the reward seems to matter for activation. Spurious rewards work nearly as well as correct ones because RLVR mostly re-activates strategies already latent in pretraining rather than teaching new ones What does reward learning actually do to model reasoning?, and models trained on deliberately corrupted reasoning traces stay just as accurate — suggesting the trace is computational scaffolding, not meaning Do reasoning traces need to be semantically correct?. Even self-generated signals like answer-span confidence can serve as the reward and improve step-by-step reasoning without any external verifier Can model confidence work as a reward signal for reasoning?. Put together, the corpus points to a counterintuitive bottom line: token efficiency isn't bought by making rewards denser — it's bought by aiming whatever reward you have at the sparse set of forking tokens that actually steer the reasoning.
Sources 10 notes
Only ~20% of tokens exhibit high entropy as pivotal reasoning decision points; RLVR primarily adjusts these forking tokens. Training exclusively on them matches or exceeds full-gradient performance, revealing that the minority carries the learning signal.
Specific tokens like "Wait" and "Therefore" show sharp spikes in mutual information with correct answers. Suppressing them harms reasoning while suppressing equal random tokens does not, and representation recycling improves accuracy 20%.
Increasing thinking tokens from ~1,100 to ~16K reduced benchmark accuracy from 87.3% to 70.3%, revealing a non-monotonic relationship where models overthink easy problems and underthink hard ones.
Greedy likelihood-preserving pruning reveals six functional token categories; symbolic computation tokens are preferentially preserved while grammar and meta-discourse are pruned first. Student models trained on these pruned chains outperform those trained on frontier-model compression.
Critique-GRPO shows that models stuck on performance plateaus can generate correct solutions when given chain-of-thought critiques, revealing that numerical rewards lack critical information about why failures occur and how to improve.
Show all 10 sources
StepWiser demonstrates that training judges to produce reasoning chains about policy reasoning—rather than classify steps—yields better judgment accuracy and data efficiency. Independent confirmation from GenPRM and ThinkPRM shows generative PRMs outperform discriminative ones with orders of magnitude less training data.
Three independent teams (RRM, RM-R1, DeepSeek-GRM) discovered that adding chain-of-thought reasoning before reward scoring enables adaptive test-time compute scaling for evaluation. Reasoning-based approaches raise the capability ceiling of reward models beyond what outcome-based evaluation achieves.
Research shows RLVR improves sampling efficiency within existing capability boundaries without expanding them. A single training example suffices for activation, and spurious rewards work nearly as well as correct ones for models with appropriate pretraining.
Models trained on systematically irrelevant traces maintain solution accuracy and sometimes improve out-of-distribution generalization, suggesting traces function as computational scaffolding rather than meaningful reasoning steps.
RLSF uses answer-span confidence to rank reasoning traces, creating synthetic preferences that strengthen step-by-step reasoning while reversing RLHF's calibration degradation—without requiring human labels or external verifiers.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- RM-R1: Reward Modeling as Reasoning
- Understanding and Mitigating Premature Confidence for Better LLM Reasoning
- Local Coherence or Global Validity? Investigating RLVR Traces in Math Domains
- Reward Reasoning Model
- Beyond Semantics: The Unreasonable Effectiveness of Reasonless Intermediate Tokens
- The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexity
- StepWiser: Stepwise Generative Judges for Wiser Reasoning
- Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning