The scorecards AI uses to learn often break in new domains — and the flaw is baked into how they score.

This explores why learned reward models fail across different domains — and the corpus points to a common root: most reward models score outputs in absolute, outcome-only terms, which doesn't travel. The clearest structural failure is calibration. Binary correctness rewards reward confident guessing because they never penalize a confident wrong answer, so a model trained this way becomes miscalibrated in exactly the way that generalizes badly to new domains where the model is less sure Does binary reward training hurt model calibration?. A scalar 'right/wrong' signal is also information-starved: it tells the model *that* it failed but not *why*, so training stalls on plateaus that no amount of numerical scaling breaks — the reward simply lacks the content needed to improve in an unfamiliar setting Can natural language feedback overcome numerical reward plateaus?.

A second failure mode is that reward models learn absolute preference labels that are tied to the task they were trained on. POLAR reframes this directly: instead of judging outputs against a fixed preference scale, it measures how close a policy is to a target policy, which is what lets its reward models transfer across task formulations where label-based RMs don't Can reward models learn by comparing policies instead of judging them?. The implication is that cross-domain failure isn't a data-volume problem — it's baked into the objective. When the reward is 'match these human labels,' the labels themselves are domain-bound.

The deeper worry is what RL-tuned rewards actually change in the model. Analyses of RLVR find that reward learning mostly sharpens sampling toward solutions the base model already contained rather than teaching genuinely new reasoning — spurious rewards work almost as well as correct ones, and a single example can trigger the effect What does reward learning actually do to model reasoning? Does RLVR actually expand what models can reason about?. That reframes 'cross-domain failure': a reward model can't reward capability the base model never had, so in a domain outside the base model's competence there's nothing for the reward to activate. This also explains why positive-only reinforcement can hurt — it concentrates probability mass and collapses the diversity you'd need to solve unfamiliar problems, whereas negative reinforcement preserves it Does negative reinforcement alone outperform full reinforcement learning?.

The corpus's proposed fixes all attack the thinness of the signal. Reasoning-based reward models add chain-of-thought before scoring, which raises the evaluator's ceiling and lets it scale test-time compute on hard, unfamiliar judgments instead of snapping to a shallow verdict Can reward models benefit from reasoning before scoring?. RLAG rewards explanation quality, not just token-level answer correctness, which is what lets domain knowledge actually internalize rather than surface-fit Can reinforcement learning embed domain knowledge more effectively than supervised fine-tuning?. And post-completion learning goes further, training the model to compute its own reward internally rather than leaning on a brittle external RM at all Can models learn to evaluate their own work during training?.

The thread worth carrying away: reward models fail across domains not mainly because they're under-trained, but because outcome-only, absolute, scalar rewards encode the wrong thing — they capture *whether* an answer matched in one domain instead of *why* it was good, and 'why' is the only part that transfers.