Why do preference models favor surface features over substance?
Preference models show systematic bias toward length, structure, jargon, sycophancy, and vagueness—features humans actively dislike. Understanding this 40% divergence reveals whether it stems from training data artifacts or architectural constraints.
Preference models — both reward models and LLM evaluators — consistently favor responses exhibiting five idiosyncratic bias features, even when these features add no substantive value. Using controlled counterfactual pairs that amplify one bias dimension while holding others constant, the miscalibration rate reaches approximately 40% divergence from human preferences.
The five dimensions:
- Length — verbosity preferred even when redundant; correlate of comprehensiveness
- Structure — bullet lists and numbered points preferred over narrative prose regardless of suitability
- Jargon — specialized terminology preferred as a proxy for expertise even when unnecessary
- Sycophancy — agreement with user preferred over neutral objectivity
- Vagueness — broad statements preferred, being less falsifiable and thus less penalized
The correlation structure reveals the mechanism. Bias features show moderate-to-strong positive correlation with model preference labels (mean r_model = +0.36) but mild negative correlation with human preference labels (mean r_human = -0.12). Models are not just slightly miscalibrated — they are systematically inverted on what these features signal.
Sycophancy divergence is the most extreme: LLM evaluators show 75-85% skew toward sycophantic responses versus ~50% for human annotators. This confirms that Can LLM judges be fooled by fake credentials and formatting? extends beyond judge biases into the reward model layer.
The downstream consequences cascade: reward models incentivize reward hacking toward these proxy features; evaluators distort benchmark conclusions; optimization toward surface properties diverges from human preferences. Counterfactual data augmentation (CDA) using synthesized contrastive examples partially corrects the miscalibration but does not eliminate it.
Inquiring lines that read this note 5
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
How can AI alignment serve diverse human preferences at scale? How do aggregate reward models systematically exclude minority user preferences? How can we distinguish genuine user preferences from measurement artifacts? What properties determine whether reward signals teach genuine reasoning?Related concepts in this collection 5
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
-
Can LLM judges be fooled by fake credentials and formatting?
Explores whether language models evaluating text fall for authority signals and visual presentation unrelated to actual content quality, and whether these weaknesses can be exploited without deep model knowledge.
judge biases and reward model biases stem from the same training data artifacts
-
Do reward models actually consider what the prompt asks?
Exploring whether standard reward models evaluate responses based on prompt context or just response quality alone. This matters because if models ignore prompts, they'll fail to align with what users actually want.
prompt insensitivity is another dimension of the same miscalibration
-
Does transformer attention architecture inherently favor repeated content?
Explores whether soft attention's tendency to over-weight repeated and prominent tokens explains sycophancy independent of training. Questions whether architectural bias precedes and enables RLHF effects.
sycophancy has both attentional (architectural) and training (data artifact) causes
-
Why do alignment methods work if they model human irrationality?
DPO and PPO-Clip succeed partly by implicitly encoding human cognitive biases like loss aversion. Does modeling irrationality explain their effectiveness better than traditional preference learning theory?
the cognitive biases that make alignment methods work are the same biases that produce preference model miscalibration; the training signal inherits human irrationality by design
-
Do therapists accurately perceive the working alliance with patients?
This research explores whether therapists' own assessments of the therapeutic relationship match what patients actually experience, especially in high-risk cases like suicidality.
compounding miscalibration: human therapists miscalibrate alliance perception at the relationship level; AI preference models miscalibrate quality assessment at the evaluation level; together they create multiple stacked measurement failures in therapeutic AI systems
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- Flattery, Fluff, and Fog: Diagnosing and Mitigating Idiosyncratic Biases in Preference Models
- Measuring Human Preferences in RLHF is a Social Science Problem
- Learning Pluralistic User Preferences through Reinforcement Learning Fine-tuned Summaries
- Verbalized Sampling: How to Mitigate Mode Collapse and Unlock LLM Diversity
- Capturing Individual Human Preferences with Reward Features
- When Large Language Models contradict humans? Large Language Models’ Sycophantic Behaviour
- Beyond Preferences in AI Alignment
- Personalization of Large Language Models: A Survey
Original note title
preference model miscalibration across five bias dimensions diverges from human preferences by 40 percent driven by training data artifacts