What makes policy discrimination scalable where preference annotation hits bottlenecks?
This explores why reframing reward modeling as *policy discrimination* (scoring how close a model's behavior is to a target) scales better than the older path of collecting human preference labels ("A is better than B") — and what specifically makes hand-annotation the bottleneck.
This explores why reframing reward modeling as *policy discrimination* — scoring how close a model is to a target policy — sidesteps the wall that hand-collected preference labels eventually hit. The clearest statement of the shift is POLAR, which redefines a reward model not as a judge that assigns an absolute "goodness" score, but as a discriminator measuring distance between a policy and a chosen target Can reward models learn by comparing policies instead of judging them?. The payoff is exactly about scale: distance is a relative signal you can generate at volume without a human writing down a preference each time, and the pre-trained discriminators transfer across task formulations rather than needing fresh labels per domain.
To see why that matters, it helps to look at what's actually wrong with preference annotation as a supply source. One line of work shows that annotation responses aren't a clean signal at all — they decompose into three different things (genuine preferences, non-attitudes, and preferences people construct on the spot), distinguishable only by whether they stay consistent across measurement conditions Do all annotation responses measure the same underlying thing?. So the bottleneck isn't only that labels are slow and expensive to gather; it's that a large fraction of them are noise or artifacts, and treating them uniformly contaminates the reward model. Policy discrimination scales partly because it stops depending on that fragile human signal.
The deeper pattern across the corpus is that the scalable methods all replace a per-example human judgment with a signal the system can manufacture itself. Test-Time RL generates rewards by majority vote across repeated samples — no ground-truth labels, no trained reward model — and bootstraps improvement from consensus Can models improve themselves using only majority voting?. Reward Reasoning Models go the other direction, raising the *quality* ceiling by letting the evaluator reason before scoring and spend more test-time compute on hard cases Can reward models benefit from reasoning before scoring?. Both share POLAR's move: shift the cost from "humans annotate" to "compute discriminates." That's the real axis of scalability here — annotation cost is linear in human effort; discrimination and self-generated reward cost is linear in compute, which you can buy.
There's a useful counterweight, though. Preference *annotation* doesn't always need to be a bottleneck if you make it adaptive instead of bulk: PReF shows that ten well-chosen active-learning questions can pin down a user's personalized reward coefficients, because the questions are selected to maximally reduce uncertainty rather than collected en masse Can user preferences be learned from just ten questions?. So the corpus actually frames two escape routes from the annotation wall — discriminate at scale without labels (POLAR, Test-Time RL), or annotate far fewer but far smarter (PReF).
The thing worth carrying away: "scalable" here doesn't mean "more data." It means changing what the reward signal is *made of* — a relative distance, a consensus vote, a reasoned comparison — so that the expensive, noisy, human-attitude-laden act of preference labeling stops sitting on the critical path. And the same research that makes that case also warns why the old path was leaky in the first place: a stated preference is three different signals wearing one label.
Sources 5 notes
POLAR reframes reward modeling as policy discrimination: RMs assign higher scores to policies similar to a chosen target, eliminating absolute preference labels. Pre-trained 1.8B-7B parameter POLAR RMs substantially outperform non-pre-trained methods and transfer across task formulations.
Behavioral science reveals that annotations contain genuine preferences, non-attitudes, and constructed preferences—distinguishable by consistency across measurement conditions. Treating them uniformly contaminates reward model training and downstream alignment.
Test-Time RL generates reward signals by majority voting across repeated samples, enabling policy improvement without ground-truth labels or trained reward models. This approach works surprisingly well because consensus answers tend to be correct, creating a bootstrapping loop where test-time compute enables training that improves the model.
Three independent teams (RRM, RM-R1, DeepSeek-GRM) discovered that adding chain-of-thought reasoning before reward scoring enables adaptive test-time compute scaling for evaluation. Reasoning-based approaches raise the capability ceiling of reward models beyond what outcome-based evaluation achieves.
PReF learns base reward functions from preference data, then uses active learning to select maximally informative questions that reduce coefficient uncertainty. Users can be personalized via inference-time reward alignment without weight modification.