Can reward model training be automated without changing feedback mechanisms?
This explores whether the reward signal that trains a model can be generated automatically — self-computed, intrinsic, or auto-weighted — while leaving the underlying feedback (human preferences, correctness checks) untouched, versus whether real gains require changing what the feedback says in the first place.
This explores whether reward model training can be put on autopilot without touching the feedback mechanism itself — and the corpus splits into two camps that, read together, sharpen the question. On the "yes, automate it" side, several lines show reward signals being manufactured without new human input. Models can learn to grade their own work during training, computing their own reward in the unused sequence space after their output and carrying zero extra inference cost Can models learn to evaluate their own work during training?. An agent's own shifting confidence toward a solution can serve as a dense, per-step reward, removing the need for a separate critic or process reward model entirely Can an agent's own beliefs guide credit assignment without critics?. And when you have several objectives, you can let the data weight them — up-weighting high-signal objectives by their variance instead of hand-tuning constants How should multiple reward objectives be weighted during training?. There's even a hint the reward content barely matters: spurious rewards can activate reasoning nearly as well as correct ones, because the training is surfacing pretraining strategies rather than teaching anything new What does reward learning actually do to model reasoning?.
Sources 9 notes
Post-Completion Learning exploits unused sequence space after model output to train self-assessment capabilities during training while maintaining zero inference cost. The model learns to compute its own reward functions, internalizing evaluation rather than relying on external reward models.
ΔBelief-RL uses log-ratios of sequential probability estimates to assign per-turn credit without critic networks or process reward models. Tested on 20 Questions, smaller models trained this way matched or exceeded prior SOTA and larger baselines while generalizing beyond training.
DVAO weights objectives by their within-group variance, automatically up-weighting high-signal objectives and suppressing noise without hyperparameter tuning. This keeps advantage magnitudes bounded and replaces fixed scalarization constants with data-driven weighting.
Research shows RLVR improves sampling efficiency within existing capability boundaries without expanding them. A single training example suffices for activation, and spurious rewards work nearly as well as correct ones for models with appropriate pretraining.
Natural feedback carries two orthogonal types of information: evaluative (how well an action performed) and directive (how it should change). Scalar rewards capture evaluation but discard directional specifics that token-level distillation can recover, making the two complementary rather than redundant.
Critique-GRPO shows that models stuck on performance plateaus can generate correct solutions when given chain-of-thought critiques, revealing that numerical rewards lack critical information about why failures occur and how to improve.
Three independent teams (RRM, RM-R1, DeepSeek-GRM) discovered that adding chain-of-thought reasoning before reward scoring enables adaptive test-time compute scaling for evaluation. Reasoning-based approaches raise the capability ceiling of reward models beyond what outcome-based evaluation achieves.
RLCF and RaR methods decompose instruction quality into verifiable sub-criteria, improving performance on benchmarks like FollowBench and HealthBench. This decomposition principle reduces overfitting to superficial artifacts that plague holistic reward models.
Binary correctness rewards incentivize high-confidence guessing because they don't penalize confident wrong answers. Adding the Brier score as a second reward term mathematically guarantees joint optimization of accuracy and calibration without trade-off.