Training a model on full sentences treats every word equally — but a few key tokens do all the actual reasoning.
Why does token-level gradient targeting matter more than aggregate loss?
This explores why *which* tokens a training signal pushes on — not just how big the total loss is — often decides whether a model actually learns to reason, and what the corpus says about targeting gradients at the token level versus optimizing an aggregate objective.
This question is really about resolution: an aggregate loss tells you *how wrong* a whole sequence was, but it averages over tokens that carry very different amounts of meaning. Several notes in the collection converge on the idea that the interesting action lives at the token level, and that smearing the signal across a sequence throws away exactly the information that matters. One striking result is that models already rank their own tokens by function — when reasoning chains are pruned greedily, symbolic-computation tokens are preserved while grammar and filler get dropped first, and students trained on these functionally-pruned chains beat those trained on blind frontier-model compression Which tokens in reasoning chains actually matter most?. If a handful of tokens do the real computing, an aggregate loss that weights them the same as connective tissue is spending gradient in the wrong places.
The flip side shows up when the signal gets too flat. When within-prompt reward variance is low, the task gradient weakens and generic regularization takes over, collapsing the policy into input-agnostic templates — filtering for high-variance prompts before updating recovers performance Why do language models collapse into generic templates?. That's the aggregate-loss failure in miniature: with nothing to sharply distinguish good from bad at the token level, the model optimizes the average and produces boilerplate. The same statistic can be turned into a weapon at two resolutions — cross-rollout variance can weight individual tokens densely *and* filter out degenerate queries, and using it at both levels trains 2–3× faster Can one statistical measure serve dual purposes in RL training?.
There's also a subtler point: aggregate objectives can actively mislead. Utility-weighted (asymmetric) losses correctly sharpen the model's *choices* but degrade its *representations*, because reweighting the loss quietly reduces the gradient signal for learning substantive features — training with a symmetric loss and adjusting predictions afterward beats optimizing the utility objective directly Can utility-weighted training loss actually harm model performance?. And when reward is assigned coarsely to whole trajectories, group-relative normalization can treat a rare accidental success as a high-advantage example, reinforcing answer-repetition and computation-skipping instead of sound reasoning Do overly hard RLVR samples actually harm model capabilities?. Coarse credit assignment doesn't just lose precision — it can reward the wrong behavior.
The corpus also pushes the lateral point that 'token level' isn't automatically the *right* level either — it's just usually finer than aggregate loss, and sometimes still too coarse. Optimizing attention distributions directly outperforms token-level RL on multimodal reasoning, because attention is where the decision actually gets made Can optimizing attention patterns improve multimodal RL better than optimizing tokens?. Predicting latents instead of tokens recovers compositional structure with exponentially fewer samples, because same-level latents are far more correlated than raw tokens Why is predicting latents more sample-efficient than tokens?. Even the famous exploration-exploitation 'trade-off' turns out to be a token-level *measurement artifact* — at the hidden-state level the two barely correlate and can improve together Is the exploration-exploitation trade-off actually fundamental?.
So the real lesson isn't 'tokens beat averages' — it's that credit assignment resolution is a design choice with teeth. Push the signal down to where the meaningful decisions happen (specific tokens, attention weights, or latents) and learning gets faster and truer; average it up into an aggregate and you either dilute the signal into templates, reward accidental shortcuts, or measure a trade-off that doesn't exist. The thing you didn't know you wanted to know: the same variance statistic that weights a token can also decide whether a whole training example is worth keeping Can one statistical measure serve dual purposes in RL training?.
Sources 8 notes
Greedy likelihood-preserving pruning reveals six functional token categories; symbolic computation tokens are preferentially preserved while grammar and meta-discourse are pruned first. Student models trained on these pruned chains outperform those trained on frontier-model compression.
When within-prompt reward variance is low, task gradients weaken and regularization dominates, pushing policies toward generic outputs. SNR-Aware Filtering—selecting high-variance prompts before updates—recovers performance across tasks and scales.
DRO reuses a single self-supervised statistic at two aggregation levels: token-level weighting in dense rewards and query-level filtering to discard degenerate comparisons. This dual use achieves 2–3× faster training with better stability on unverifiable tasks.
Asymmetric loss functions correctly incentivize choosing but degrade representation learning by reducing gradient signals for substantive feature acquisition. Training with symmetric loss then adjusting predictions post-hoc outperforms direct utility-weighted training on the same utility objective.
Training on nearly-impossible problems causes models to learn degenerate shortcuts rather than genuine reasoning, and these shortcuts contaminate pre-existing capabilities. Group-relative normalization treats rare accidental successes as high-advantage trajectories, reinforcing answer repetition and computation-skipping instead of sound reasoning patterns.
Show all 8 sources
Reinforced Attention Learning treats attention patterns as the primary policy target rather than token sequences. Direct optimization of information allocation shows stronger gains on visual reasoning than standard RLHF, because attention is where the actual decision happens.
A formal sample-complexity analysis proves latent-level self-supervision (data2vec/JEPA style) recovers compositional structure with samples constant in hierarchy depth, while token-level learning requires exponential samples—because same-level latents are far more correlated than raw tokens.
Hidden-state analysis using Effective Rank metrics shows near-zero correlation between exploration and exploitation, revealing the trade-off emerges only at token level. VERL demonstrates simultaneous enhancement achieving 21.4% accuracy gains on Gaokao 2024.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- DVAO: Dynamic Variance-adaptive Advantage Optimization for Multi-reward Reinforcement Learning
- Mechanistically Interpreting the Role of Sample Difficulty in RLVR for LLMs
- Echo Chamber: RL Post-training Amplifies Behaviors Learned in Pretraining
- Farther the Shift, Sparser the Representation: Analyzing OOD Mechanisms in LLMs
- A Survey on Test-Time Scaling in Large Language Models: What, How, Where, and How Well?
- RL-PLUS: Countering Capability Boundary Collapse of LLMs in Reinforcement Learning with Hybrid-policy Optimization
- ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models
- Direct Reasoning Optimization: Token-Level Reasoning Reflectivity Meets Rubric Gates for Unverifiable Tasks