How do different training objectives shift whether models over-predict or under-predict?
This explores how the choice of training objective — what gets rewarded versus penalized — systematically tilts a model toward answering too eagerly (over-predicting) or holding back too much (under-predicting), rather than treating that bias as a random quirk.
This explores how the choice of training objective tilts a model toward over- or under-predicting — and the corpus's sharpest claim is that the direction of the error is a fingerprint of which objective dominated training, not a single dial you tune. The clearest case: models trained for reasoning learn to *over-answer*, because abstaining ('I don't know') is never rewarded, while models trained for safety learn to *over-abstain*, refusing even harmless questions Does training objective determine which direction models fail at abstention?. Same architecture, opposite failure — and the deciding factor is what the reward signal valued. That reframes calibration from 'a bug to fix' into 'a characteristic signature you inherit from your training recipe.'
The mechanism behind over-prediction shows up most cleanly in reward design. Binary correctness rewards — right answer good, wrong answer bad — quietly teach the model to guess confidently, because a confident wrong answer costs exactly the same as a hesitant one. There's no penalty for bravado. Adding the Brier score (which scores not just whether you're right but how confident you should have been) provably restores calibration without sacrificing accuracy Does binary reward training hurt model calibration?. So the over-prediction isn't the model being reckless — it's the objective failing to price uncertainty.
Push the reward in the wrong way and the bias gets worse rather than better. Training on problems that are nearly impossible for the model warps it toward degenerate shortcuts — repeating answers, skipping computation — because the rare accidental success gets treated as a high-value trajectory worth reinforcing Do overly hard RLVR samples actually harm model capabilities?. And there's a subtler twist for decision-focused training: weighting the loss by how costly a mistake is (asymmetric loss) does sharpen the model's *choices*, but it weakens the underlying *learning* by muting the gradient signal for genuine feature acquisition. The counterintuitive fix is to train with a plain symmetric objective and then adjust the predictions afterward — separating 'learn the world' from 'decide what to do' beats baking the bias into the weights Can utility-weighted training loss actually harm model performance?.
That 'adjust afterward' theme runs deeper than calibration. Several notes converge on the idea that *how aggressively* an objective rewrites the base model determines how distorted its predictions become. RL post-training tends to collapse onto a single dominant output format from pretraining within the first epoch, suppressing alternatives the model could have used Does RL training collapse format diversity in pretrained models?. Direct fine-tuning corrupts knowledge stored in lower layers, whereas decoding-time proxy tuning closes most of the alignment gap while leaving those weights — and the knowledge — intact Can decoding-time tuning preserve knowledge better than weight fine-tuning?. Staying close to the base distribution (low KL drift) even preserves the model's ability to keep learning later Does staying close to the base model preserve learning ability?. The pattern: objectives that yank the model far from its starting point trade prediction quality and plasticity for whatever they're optimizing.
The thing you didn't know you wanted to know: over- and under-prediction aren't really about the model being overconfident or timid — they're about what the objective forgot to penalize. A reasoning reward forgot to value silence; a binary reward forgot to value calibrated doubt; a utility-weighted loss forgot that good decisions need good representations underneath. Once you see calibration failures as the negative space of the reward function, the fix is usually to add the missing term (Brier score) or move the adjustment out of training entirely (proxy tuning, post-hoc correction) rather than to train harder on the same lopsided signal.
Sources 7 notes
Reasoning-trained models under-abstain and overanswer because abstention is unrewarded, while safety-trained models over-abstain and refuse benign questions. This reveals calibration is not a single fixable axis but a characteristic failure signature that depends on which objective dominated training.
Binary correctness rewards incentivize high-confidence guessing because they don't penalize confident wrong answers. Adding the Brier score as a second reward term mathematically guarantees joint optimization of accuracy and calibration without trade-off.
Training on nearly-impossible problems causes models to learn degenerate shortcuts rather than genuine reasoning, and these shortcuts contaminate pre-existing capabilities. Group-relative normalization treats rare accidental successes as high-advantage trajectories, reinforcing answer repetition and computation-skipping instead of sound reasoning patterns.
Asymmetric loss functions correctly incentivize choosing but degrade representation learning by reducing gradient signals for substantive feature acquisition. Training with symmetric loss then adjusting predictions post-hoc outperforms direct utility-weighted training on the same utility objective.
Controlled experiments show RL consistently amplifies one format distribution from pretraining within the first epoch while collapsing alternatives. The winning format depends on model scale, not necessarily performance, and is largely hidden when starting from proprietary pretrained models.
Proxy-tuning closes 88-91% of the alignment gap while surpassing direct fine-tuning on knowledge tasks by leaving base model weights untouched. Direct fine-tuning corrupts knowledge storage in lower layers, whereas proxy-tuning applies distributional shifts that primarily affect reasoning and style.
FST-trained models stay up to 70% closer to their base distribution than parameter-only RL, and this reduced drift preserves the model's ability to learn subsequent tasks effectively. Parameter-only approaches stall when task domains change, while low KL drift enables sustained adaptation.