Can one AI grader really stand in for everyone's taste, or does it quietly flatten minority views into an average?
Can a single AI judge capture diverse human preferences or does it collapse them?
This explores whether one scoring model, whether a reward model or a judge, can stand in for many people's different tastes, or whether it flattens them into a single average.
This explores whether one scoring model can stand in for many people's different tastes, or whether it flattens them into an average. The corpus's clearest answer is that it collapses them, and there is a proof. Can a single reward model represent diverse human preferences? shows that fitting one reward model to everyone's pooled preferences silently erases minority viewpoints. It's a mathematical limit of the single-score design, so better tuning can't fix it. The proposed fix is to learn a mixture of preference groups and optimize for the worst-off group, borrowing a rule from social choice theory. That paper is about reward models that train AI, not judges that grade outputs. Both compress many opinions into one number, though, so the same logic applies.
The collapse also happens inside a single person. Can attention mechanisms reveal which user taste explains each recommendation? finds that even one user is better modeled as several personas, weighted differently depending on the item in front of them. A judge that assumes one stable taste per person is too flat before it even reaches the group level. The labels feeding a judge are also messier than they look. Do all annotation responses measure the same underlying thing? argues that annotations mix genuine preferences, non-attitudes (answers given with no real opinion), and preferences constructed on the spot. A single judge averages over signals that were never the same kind of thing. It can lose real minority tastes while faithfully learning noise.
The corpus also points to a way out that doesn't require one judge per person. Can user preferences be learned from just ten questions? learns a shared set of base reward functions, then finds a person's mix with about ten well-chosen questions, without retraining the model. That gives one common set of building blocks and many personal judges. Collapse also isn't universal. Does preference tuning always reduce diversity the same way? finds that preference tuning narrows outputs in code, where convergence on the correct answer is the goal, but widens them in creative writing. A single judge does the least harm where there's a right answer and the most where taste is the point.
The last piece is a caution about how good a single judge can look. Can AI predict social norms better than humans? reports a model that predicts what people find socially appropriate better than any individual human, yet can't take part in the community processes that create those norms. High agreement with the average is not the same as representing a living spread of views. On the judge side, Can agents evaluate AI outputs more reliably than language models? cuts judge shift from 31% to 0.27% by having agents gather evidence. That buys consistency, but the corpus doesn't test whether it buys diversity. A very stable judge can be stably one-sided.
Sources 7 notes
MaxMin-RLHF proves an impossibility result: fitting one reward model to aggregated preferences silently erases minority viewpoints. The solution is learning a mixture of preference distributions and optimizing a MaxMin objective from social choice theory to protect the worst-off groups.
AMP-CF represents each user as multiple latent personas weighted dynamically by candidate item. This makes recommendations both diverse and interpretable—each suggestion traces to the specific persona preference it satisfies—without requiring post-hoc reranking.
Behavioral science reveals that annotations contain genuine preferences, non-attitudes, and constructed preferences—distinguishable by consistency across measurement conditions. Treating them uniformly contaminates reward model training and downstream alignment.
PReF learns base reward functions from preference data, then uses active learning to select maximally informative questions that reduce coefficient uncertainty. Users can be personalized via inference-time reward alignment without weight modification.
RLHF reduces lexical-syntactic diversity in code generation but increases it in creative writing. The direction depends on what each domain incentivizes: code rewards convergence toward correct solutions, while creative writing rewards stylistic distinctiveness.
Show all 7 sources
GPT-4.5 outperforms all individual humans at predicting social appropriateness, yet structurally cannot enter the community processes that establish and validate norms. This reveals a critical gap between pattern-matching and authentic participation in knowledge-making.
Eight-module agentic evaluation achieved 0.27% judge shift versus 31% for LLM-as-a-Judge on complex tasks. However, the memory module cascaded errors, revealing that agentic systems need error isolation mechanisms to maintain gains.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Capturing Individual Human Preferences with Reward Features
- Measuring Human Preferences in RLHF is a Social Science Problem
- MaxMin-RLHF: Alignment with Diverse Human Preferences
- Learning Pluralistic User Preferences through Reinforcement Learning Fine-tuned Summaries
- Beyond Preferences in AI Alignment
- Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learning
- Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustment
- SOTOPIA: Interactive Evaluation for Social Intelligence in Language Agents