Can aggregate reward models satisfy genuinely disagreeing users?
When users have conflicting preferences, do aggregate reward models face an impossible choice between satisfying majorities or sampling proportionally? What does this reveal about RLHF deployment?
A clean argument for why aggregate reward models cannot serve disagreement-heavy tasks. Consider a subjective question where 51% of the target audience prefer answer A and 49% prefer answer B. With a single reward model trained on aggregated preferences, the deployment has exactly two options. Pick A as the preferred answer: 49% of users are unhappy 100% of the time. Sample A and B proportionally to their preference rates: 100% of users are unhappy approximately half the time. Both options are unsatisfactory.
The structural problem is that aggregate reward models compress preference distributions into single scalars (or single rankings) that cannot represent disagreement. They reward what the majority prefers and incidentally suppress what the minority prefers. For tasks with high consensus this is fine — the majority preference is everyone's preference. For tasks with genuine disagreement — subjective evaluations, value-laden topics, creative judgment, cultural-context-dependent choices — aggregate models systematically exclude the minority view.
This is not a quality problem with current reward models. It is a representational problem with the aggregation step itself. Even a perfect aggregate reward model would face this dilemma. The fix has to operate at a different level: reward models that can be specialized to individual users (or to user groups whose preferences cluster) rather than averaged across the population.
The implication extends beyond personalization. Whenever a system is deployed against a heterogeneous user base with genuinely divergent preferences, the standard "train one model to satisfy everyone" architecture is incompatible with satisfying anyone fully. The right architecture either splits per-user (personalization) or splits per-cluster (group-level adaptation). Aggregate reward modeling becomes appropriate only when the underlying preferences are actually unimodal — and that is a stronger assumption than RLHF deployments typically test.
Inquiring lines that read this note 46
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
How do different reward signals affect model reasoning and calibration differently?- How does RLHF reward structure incentivize agreement over accuracy?
- Does user preference for confirmation override model capability for disagreement?
- Can proper scoring rules fix RLVR's degradation on disagreement prediction?
- How can consistency across measurement conditions identify genuine versus constructed preferences?
- Do high-disagreement items signal contested values or measurement noise?
- What consistency tests could distinguish constructed from genuine preferences?
- What makes preference distributions unimodal versus genuinely disagreement-heavy?
- When does clustering users by preference overcome the aggregation dilemma?
- Why does preference measurement validity matter more than aggregation methods?
- What validity threats exist in crowdsourced preference signals?
- How can developers balance multiple conflicting fairness goals simultaneously?
- Why does preference measurement validity matter before any aggregation?
- Does majority voting reliably signal correctness without risking reward hacking?
- What happens when majority voting converges to a single answer?
- Why do standard preference alignment methods fail at the individual user level?
- What preference dimensions do base reward functions typically capture?
- What makes minority preferences disappear in aggregated single-distribution reward models?
- How do aggregate reward models fail to capture minority user preferences?
- Can reward models distinguish between personal preference and community consensus?
- How do binary comparisons constrain reward scale in multi-user preference learning?
- Why does single-reward RLHF fail to represent diverse human preferences?
- How do aggregate reward models systematically exclude minority perspectives?
- How do aggregate reward models systematically exclude minority preferences?
- Does reward model training data quality determine fair preference aggregation outcomes?
- Can social welfare functions avoid unjustified assumptions about comparing people's preferences?
- Can reward models be personalized if annotators lack stable preferences?
- How do personalized reward models avoid excluding minority viewpoints?
- Can user preferences be represented as linear reward combinations?
- Can elicited user responses measure true preferences or just elicitation artifacts?
- Are goal-satisfaction discounts and welfare debits measured in commensurable units?
- Do welfare goals and veto-resistance align or pull in opposite directions?
- Can additive welfare aggregation justify removing minority override rights?
- Why do welfare goals that sum welfare levels keep the veto gap open?
- Why does additive aggregation create asymmetry between welfare and veto preservation?
- How might aggregative welfare goals justify overriding minority preferences?
Related concepts in this collection 3
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
-
Does preference data need more raters than examples?
Pairwise preference data violates the i.i.d. assumption because preferences vary across raters. Does this mean PAC bounds for reward models depend on rater diversity rather than just sample size?
same paper, the theoretical foundation
-
Does personalizing reward models amplify user echo chambers?
Personalized reward models solve the minority-preference problem but may introduce new risks by reinforcing existing user beliefs and narrowing exposure to diverse viewpoints.
same paper, the tension with personalization
-
Can user preferences be learned from just ten questions?
Explores whether adaptive question selection can efficiently infer user-specific reward coefficients without historical data or fine-tuning. This matters for scaling personalization without per-user model updates.
adjacent: the technical solution to the aggregation problem
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- Measuring Human Preferences in RLHF is a Social Science Problem
- Capturing Individual Human Preferences with Reward Features
- Beyond Preferences in AI Alignment
- Can Large Language Models Capture Human Annotator Disagreements?
- Flattery, Fluff, and Fog: Diagnosing and Mitigating Idiosyncratic Biases in Preference Models
- Information-Theoretic Reward Decomposition for Generalizable RLHF
- RLHF Workflow: From Reward Modeling to Online RLHF
- Self-Improving Model Steering
Original note title
aggregate reward models systematically exclude minority preferences — the dilemma of preferred answer or proportional sampling is a structural failure of one-size-fits-all RLHF