How do reward models as policy discriminators differ from labeled preferences?
This explores POLAR's reframing of reward modeling — scoring how close a policy sits to a target rather than judging answers against human preference labels — and what that shift buys you over the standard labeled-preference approach.
This explores how reward models can be rebuilt as *policy discriminators* — systems that measure distance from a target policy — instead of the usual recipe of training on human-labeled preference pairs. The cleanest statement of the idea is POLAR Can reward models learn by comparing policies instead of judging them?, which flips the question. A conventional reward model learns absolute judgments: 'this answer is better than that one,' grounded in human labels. POLAR instead asks 'how similar is this policy to a chosen target?' and assigns higher scores to closer policies. That sidesteps absolute preference labels entirely, and the payoff is transfer — pretrained discriminators in the 1.8B–7B range beat non-pretrained methods and carry across task formulations rather than being welded to one labeling scheme.
Why would you want to escape labels at all? Because labels are noisier than they look. One striking finding is that annotation responses aren't a single signal — they decompose into genuine preferences, non-attitudes, and constructed-on-the-spot preferences, distinguishable only by whether they stay consistent across measurement conditions Do all annotation responses measure the same underlying thing?. Treat them all as ground truth and you contaminate the reward model. A discriminator that measures distance-from-target inherits the quality of its target policy instead of the quality of a crowd's momentary clicks, which is a different — and often cleaner — failure surface.
The deeper theme across the corpus is that the *scalar preference label* is a lossy container. Agent feedback, for instance, carries two orthogonal things at once: evaluative information (how good was this?) and directive information (how should it change?), and a single scalar reward can only hold the first Can scalar rewards capture all the information in agent feedback?. Other notes route around labels in their own ways: test-time RL manufactures reward from majority vote across samples with no labels at all Can models improve themselves using only majority voting?, and ΔBelief-RL turns the agent's own shifting confidence toward a solution into dense per-step credit, no preference annotations or critic networks required Can an agent's own beliefs guide credit assignment without critics?. Discrimination-from-target is one member of this larger family: ways to get a training signal without asking humans to rank outputs.
There's also a personalization angle worth knowing. The averaging that aggregate preference labels perform is quietly protective — it washes out individual quirks. Specialize a reward model per user and you remove that buffer, and the system can learn sycophancy and harden echo chambers, exactly the way recommender systems do Does personalizing reward models amplify user echo chambers?. A target-policy discriminator reframes 'whose preferences' as 'which target,' which makes that choice explicit rather than hidden inside a label distribution.
If you want to go further, two notes complicate the picture in useful directions: reward models that *reason* before scoring raise their own capability ceiling beyond outcome-only evaluation Can reward models benefit from reasoning before scoring?, and binary correctness rewards — the simplest labels of all — provably wreck calibration unless you bolt on a proper scoring rule Does binary reward training hurt model calibration?. Together they suggest the interesting frontier isn't 'discriminator vs. labels' but how much structure you let the reward signal carry.
Sources 8 notes
POLAR reframes reward modeling as policy discrimination: RMs assign higher scores to policies similar to a chosen target, eliminating absolute preference labels. Pre-trained 1.8B-7B parameter POLAR RMs substantially outperform non-pre-trained methods and transfer across task formulations.
Behavioral science reveals that annotations contain genuine preferences, non-attitudes, and constructed preferences—distinguishable by consistency across measurement conditions. Treating them uniformly contaminates reward model training and downstream alignment.
Natural feedback carries two orthogonal types of information: evaluative (how well an action performed) and directive (how it should change). Scalar rewards capture evaluation but discard directional specifics that token-level distillation can recover, making the two complementary rather than redundant.
Test-Time RL generates reward signals by majority voting across repeated samples, enabling policy improvement without ground-truth labels or trained reward models. This approach works surprisingly well because consensus answers tend to be correct, creating a bootstrapping loop where test-time compute enables training that improves the model.
ΔBelief-RL uses log-ratios of sequential probability estimates to assign per-turn credit without critic networks or process reward models. Tested on 20 Questions, smaller models trained this way matched or exceeded prior SOTA and larger baselines while generalizing beyond training.
Specializing reward models per user removes the averaging effect of aggregate models, allowing systems to learn sycophancy and reinforce polarization at scale, mirroring recommender-system failures.
Three independent teams (RRM, RM-R1, DeepSeek-GRM) discovered that adding chain-of-thought reasoning before reward scoring enables adaptive test-time compute scaling for evaluation. Reasoning-based approaches raise the capability ceiling of reward models beyond what outcome-based evaluation achieves.
Binary correctness rewards incentivize high-confidence guessing because they don't penalize confident wrong answers. Adding the Brier score as a second reward term mathematically guarantees joint optimization of accuracy and calibration without trade-off.