Can aggregate reward models satisfy genuinely disagreeing users?
When users have conflicting preferences, do aggregate reward models face an impossible choice between satisfying majorities or sampling proportionally? What does this reveal about RLHF deployment?
A clean argument for why aggregate reward models cannot serve disagreement-heavy tasks. Consider a subjective question where 51% of the target audience prefer answer A and 49% prefer answer B. With a single reward model trained on aggregated preferences, the deployment has exactly two options. Pick A as the preferred answer: 49% of users are unhappy 100% of the time. Sample A and B proportionally to their preference rates: 100% of users are unhappy approximately half the time. Both options are unsatisfactory.
The structural problem is that aggregate reward models compress preference distributions into single scalars (or single rankings) that cannot represent disagreement. They reward what the majority prefers and incidentally suppress what the minority prefers. For tasks with high consensus this is fine — the majority preference is everyone's preference. For tasks with genuine disagreement — subjective evaluations, value-laden topics, creative judgment, cultural-context-dependent choices — aggregate models systematically exclude the minority view.
This is not a quality problem with current reward models. It is a representational problem with the aggregation step itself. Even a perfect aggregate reward model would face this dilemma. The fix has to operate at a different level: reward models that can be specialized to individual users (or to user groups whose preferences cluster) rather than averaged across the population.
The implication extends beyond personalization. Whenever a system is deployed against a heterogeneous user base with genuinely divergent preferences, the standard "train one model to satisfy everyone" architecture is incompatible with satisfying anyone fully. The right architecture either splits per-user (personalization) or splits per-cluster (group-level adaptation). Aggregate reward modeling becomes appropriate only when the underlying preferences are actually unimodal — and that is a stronger assumption than RLHF deployments typically test.
Inquiring lines that read this note 34
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
Does RLHF training sacrifice accuracy and grounding for user agreement?- How does RLHF reward structure incentivize agreement over accuracy?
- What unmeasured side channels emerge from RLHF preference optimization?
- How can consistency across measurement conditions identify genuine versus constructed preferences?
- What consistency tests could distinguish constructed from genuine preferences?
- What makes preference distributions unimodal versus genuinely disagreement-heavy?
- Why does preference measurement validity matter more than aggregation methods?
- Why does preference measurement validity matter before any aggregation?
- Does majority voting reliably signal correctness without risking reward hacking?
- What happens when majority voting converges to a single answer?
- What distinguishes actual social disagreement from distributional uncertainty in LLM outputs?
- Can aggregate survey realism coexist with unreliable fine-grained effects?
- Why do standard preference alignment methods fail at the individual user level?
- How can developers balance multiple conflicting fairness goals simultaneously?
- Can reward models be personalized if annotators lack stable preferences?
- What makes minority preferences disappear in aggregated single-distribution reward models?
- How do personalized reward models avoid excluding minority viewpoints?
- When does clustering users by preference overcome the aggregation dilemma?
- Can citizen assemblies and value pluralism replace single utility optimization?
- How do aggregate reward models fail to capture minority user preferences?
- Can user preferences be represented as linear reward combinations?
- Can reward models distinguish between personal preference and community consensus?
- Why does single-reward RLHF fail to represent diverse human preferences?
- How do aggregate reward models systematically exclude minority perspectives?
- What validity threats exist in crowdsourced preference signals?
- How do aggregate reward models systematically exclude minority preferences?
- What preference dimensions do base reward functions typically capture?
- How do binary comparisons constrain reward scale in multi-user preference learning?
Related concepts in this collection 3
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
-
Does preference data need more raters than examples?
Pairwise preference data violates the i.i.d. assumption because preferences vary across raters. Does this mean PAC bounds for reward models depend on rater diversity rather than just sample size?
same paper, the theoretical foundation
-
Does personalizing reward models amplify user echo chambers?
Personalized reward models solve the minority-preference problem but may introduce new risks by reinforcing existing user beliefs and narrowing exposure to diverse viewpoints.
same paper, the tension with personalization
-
Can user preferences be learned from just ten questions?
Explores whether adaptive question selection can efficiently infer user-specific reward coefficients without historical data or fine-tuning. This matters for scaling personalization without per-user model updates.
adjacent: the technical solution to the aggregation problem
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- Measuring Human Preferences in RLHF is a Social Science Problem
- Capturing Individual Human Preferences with Reward Features
- Beyond Preferences in AI Alignment
- Can Large Language Models Capture Human Annotator Disagreements?
- Flattery, Fluff, and Fog: Diagnosing and Mitigating Idiosyncratic Biases in Preference Models
- Information-Theoretic Reward Decomposition for Generalizable RLHF
- RLHF Workflow: From Reward Modeling to Online RLHF
- Self-Improving Model Steering
Original note title
aggregate reward models systematically exclude minority preferences — the dilemma of preferred answer or proportional sampling is a structural failure of one-size-fits-all RLHF