What makes reasoning tokens identifiable within rollout groups for better rewards?
This explores how RL training picks out which tokens inside a group of sampled rollouts actually carry the reasoning signal — so that credit (reward) lands on the moments that mattered rather than being smeared across the whole trajectory.
This explores how RL training picks out which tokens inside a group of sampled rollouts actually carry the reasoning — the question behind a lot of recent reward-shaping work, because outcome-only rewards tell you a trajectory was right without telling you *where* it earned that. The corpus converges on two intuitions: a few tokens matter far more than the rest, and the group of rollouts itself is the instrument that reveals which ones.
Start with the claim that reasoning is concentrated, not diffuse. High-entropy "forking" tokens turn out to be the decision points where a trajectory could branch, and RLVR mostly adjusts those — training on just the ~20% high-entropy tokens matches or beats full-gradient updates Do high-entropy tokens drive reasoning model improvements?. From a different angle, specific words like "Wait" and "Therefore" spike in mutual information with the correct answer; suppress them and accuracy falls, suppress an equal number of random tokens and nothing happens Do reflection tokens carry more information about correct answers?. So "identifiable" has a measurable signature — entropy and information content both flag the same sparse, pivotal minority.
But entropy is a property of a single rollout. What makes tokens identifiable *within a group* is comparison across rollouts that share a context. Tree-search rollouts branch from a shared prefix, then compare sibling subtrees: because siblings differ only after the fork, the outcome gap between them is automatically attributable to that step — turning a trajectory-level reward into step-level process supervision with no separate reward model or human annotation Can tree structure alone convert outcome rewards into process supervision?. Shared-prefix trees also stretch the budget: branching produces more *distinct* trajectories per token than independent sampling, which sharpens the advantage estimates the comparison relies on Can shared-prefix trees reduce redundancy in agent rollouts?. The structure of the group is doing the identification.
A cleaner version of the same idea uses variance directly. DRO reuses one self-supervised statistic — how much rollouts disagree at a given point — to both weight tokens densely and filter out degenerate queries where every rollout agrees and there's nothing to learn, getting 2–3× faster training on unverifiable tasks Can one statistical measure serve dual purposes in RL training?. Where rollouts diverge is exactly where the model is making a consequential choice; that divergence is the reward signal.
The last piece is *who* decides a token deserves reward. The trend is away from a classifier that scores a step and toward a judge that reasons about the step: generative stepwise judges (StepWiser, GenPRM, ThinkPRM) outperform discriminative reward models with orders of magnitude less data Can judges that reason about reasoning outperform classifier rewards?, and reward models themselves improve when allowed to think before scoring, scaling test-time compute on the evaluation side Can reward models benefit from reasoning before scoring?. Put together, the corpus suggests reasoning tokens become identifiable through a convergence of signals — entropy, mutual information, cross-rollout variance, and sibling comparison — and the better rewards come from a judge sophisticated enough to act on them.
Sources 7 notes
Only ~20% of tokens exhibit high entropy as pivotal reasoning decision points; RLVR primarily adjusts these forking tokens. Training exclusively on them matches or exceeds full-gradient performance, revealing that the minority carries the learning signal.
Specific tokens like "Wait" and "Therefore" show sharp spikes in mutual information with correct answers. Suppressing them harms reasoning while suppressing equal random tokens does not, and representation recycling improves accuracy 20%.
Tree-GRPO uses branching structure to transform trajectory-level outcome rewards into step-level preference signals through sibling subtree comparison, eliminating the need for separate process reward models or step-level annotation while scaling with computational budget.
Tree-structured rollouts that branch from shared prefixes produce more distinct trajectories within a fixed token budget than independent chain sampling. This improves advantage estimation statistics and enables longer-horizon tasks within the same compute constraint.
DRO reuses a single self-supervised statistic at two aggregation levels: token-level weighting in dense rewards and query-level filtering to discard degenerate comparisons. This dual use achieves 2–3× faster training with better stability on unverifiable tasks.
StepWiser demonstrates that training judges to produce reasoning chains about policy reasoning—rather than classify steps—yields better judgment accuracy and data efficiency. Independent confirmation from GenPRM and ThinkPRM shows generative PRMs outperform discriminative ones with orders of magnitude less training data.
Three independent teams (RRM, RM-R1, DeepSeek-GRM) discovered that adding chain-of-thought reasoning before reward scoring enables adaptive test-time compute scaling for evaluation. Reasoning-based approaches raise the capability ceiling of reward models beyond what outcome-based evaluation achieves.