If you blend two noisy signals by trusting each in proportion to its reliability, is the result cleaner than your best one?
Does inverse-variance denoising reduce variance below either reward stream alone?
This explores whether combining two noisy reward signals, each weighted by how much you trust it (inverse-variance weighting), gives a cleaner signal than the better of the two alone.
This explores whether combining two noisy reward signals, each weighted by how much you trust it (inverse-variance weighting), gives a cleaner signal than the better of the two alone. The collection has no note that tests this directly, so the core answer is textbook statistics rather than something the corpus demonstrates. If both streams are unbiased and their errors are independent, the combined variance is 1/(1/σ₁² + 1/σ₂²). That is always below the smaller of the two variances, so even the noisier stream helps a little. The corpus is useful for the conditions that decide whether this holds.
The first condition is that the noise must be independent. Differential attention is the mirror image of the idea: Can subtracting attention maps remove noise from context? subtracts two softmax maps to cancel the noise floor they share, like noise-cancelling headphones. Averaging removes independent noise, while subtraction removes shared noise. If your two reward streams make the same mistakes, weighting them cleverly won't help. That may be why Can peer models replace external judges for reward signals? finds that diverse peer cohorts beat single-model self-rewarding, which suffers bias and collapse. My reading is that diversity is what makes the errors independent enough to average away.
The second condition is that both streams must estimate the same quantity, and variance reduction can't fix bias. Can scalar rewards capture all the information in agent feedback? argues that evaluative and directive feedback are orthogonal and complementary, not two noisy copies of one signal. Blending them is about coverage, not noise. [[binary-reward-rl-provably-degrades-calibration-adding-a-proper-scoring-rule-as-a]} shows the same thing with correctness plus a Brier score. Binary rewards push models toward confident guessing, which is a systematic tilt. Averaging in more of the same reward wouldn't remove it. A second term with a different target does.
The closest the collection gets to using variance as a trust signal is Can one statistical measure serve dual purposes in RL training?. It reuses cross-rollout variance to weight tokens and to filter out degenerate queries, reporting 2–3× faster training. That is one stream with variance as a weight, not two streams fused. Do overly hard RLVR samples actually harm model capabilities? is a caution: group-relative normalization turns rare accidental successes into high-advantage trajectories and reinforces shortcuts. Rescaling a signal by its spread can promote a fluke as easily as a real improvement.
So the variance drop is guaranteed on paper, but only for unbiased, independent, like-for-like streams. Whether reward streams in practice meet those conditions is not something the collection has tested.
Sources 6 notes
DIFF Transformer computes attention as the difference between two softmax maps, mechanically canceling the shared noise floor while preserving differential signal—analogous to noise-cancelling headphones. This produces sparse, focused attention patterns without sacrificing efficiency.
Co-RL trains decoupled models using peer predictions as rewards, avoiding the bias and collapse of self-generated feedback. Heterogeneous cohorts consistently improve reasoning across benchmarks and often match ground-truth supervised training.
Natural feedback carries two orthogonal types of information: evaluative (how well an action performed) and directive (how it should change). Scalar rewards capture evaluation but discard directional specifics that token-level distillation can recover, making the two complementary rather than redundant.
Binary correctness rewards incentivize high-confidence guessing because they don't penalize confident wrong answers. Adding the Brier score as a second reward term mathematically guarantees joint optimization of accuracy and calibration without trade-off.
DRO reuses a single self-supervised statistic at two aggregation levels: token-level weighting in dense rewards and query-level filtering to discard degenerate comparisons. This dual use achieves 2–3× faster training with better stability on unverifiable tasks.
Show all 6 sources
Training on nearly-impossible problems causes models to learn degenerate shortcuts rather than genuine reasoning, and these shortcuts contaminate pre-existing capabilities. Group-relative normalization treats rare accidental successes as high-advantage trajectories, reinforcing answer repetition and computation-skipping instead of sound reasoning patterns.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Can Large Reasoning Models Self-Train?
- Reward Reasoning Model
- ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models
- Co-RL: Unsupervised Reasoning Emerges from Diverse Cohort in Multi-agent RL
- Differential Transformer
- Mechanistically Interpreting the Role of Sample Difficulty in RLVR for LLMs
- Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?
- Reinforcement Learning for Reasoning in Large Language Models with One Training Example