Splitting AI training into step-by-step feedback plus a hard pass/fail gate beats one blunt score the model can game.

This explores why splitting the training signal — a dense, step-by-step reward paired with a hard gate — outperforms one fixed reward that has to serve every purpose at once. The cleanest statement of the mechanism comes from DRO, which separates optimization from feasibility: instead of turning a rubric into a reward you can maximize, it uses the rubric as a gate that accepts or rejects whole answer groups, then lets token-level rewards optimize only within the answers that already passed Can rubrics and dense rewards work together without hacking?. The reason this matters is that a single reward invites gaming: if the rubric score is something you can partially satisfy for partial credit, the model finds cheap ways to inflate it. As a hard constraint, the rubric can't be hacked — you're either inside the valid region or you're not — and all the delicate optimization happens somewhere it can't corrupt the definition of "correct."

A single fixed reward also throws away information. Natural feedback actually carries two orthogonal signals: evaluative (how good was this?) and directive (which way should it change?). A scalar reward captures the first and discards the second entirely Can scalar rewards capture all the information in agent feedback?. That lost directive channel is exactly why models plateau on pure numerical rewards — they know they failed but not why or how to fix it — and why layering in chain-of-thought critique lets stuck models suddenly produce correct solutions Can natural language feedback overcome numerical reward plateaus?. Dense rewards recover some of that lost information by scoring the path, not just the destination.

The density part earns its keep most when the outcome signal is too sparse to learn from. Step-wise expert-similarity rewards give a learning signal at every step even when every rollout ultimately fails — bridging rigid imitation (SFT) and sparse outcome-only rewards (RLVR) — which is what lets small models learn hard reasoning they'd otherwise never get traction on Can step-wise expert rewards help small models learn hard reasoning?. A single outcome reward on an all-failing batch is a flat zero: no gradient, nothing to climb.

There's a second, subtler benefit to adding a constraint as a separate term rather than folding everything into accuracy: it fixes pathologies the single reward silently creates. Binary correctness rewards, for instance, quietly destroy calibration, because a confident wrong answer costs exactly the same as a hesitant one — so the model learns to bluster. Adding a proper scoring rule (the Brier score) as a second reward term provably restores joint optimization of accuracy and calibration with no trade-off Does binary reward training hurt model calibration?. The general pattern: one reward can only point in one direction, so anything it doesn't measure gets optimized away.

What's worth sitting with is the ceiling on all of this. RLVR — the paradigm most of these dense-reward tricks live inside — mostly sharpens sampling toward solutions the base model already contained rather than teaching genuinely new reasoning; base models can still beat RLVR-tuned ones at high sample counts Does RLVR actually expand what models can reason about? What does reward learning actually do to model reasoning?. So the honest framing of "dense rewards plus hard constraints" isn't that richer signals unlock new capability — it's that they extract far more of the capability already latent in the model, cleanly and without hacking, than a single blunt reward ever could.