When human reviewers disagree on what counts as a good AI answer, does forcing them to agree erase the very judgment that made their input valuable?
How do rater disagreements and consensus processes affect tacit knowledge production in RLHF?
This explores what happens to the hard-to-articulate judgment that human raters put into RLHF when they disagree with each other, and whether averaging or settling those disagreements keeps that judgment or throws it away.
This explores what happens to the hard-to-articulate judgment that human raters put into RLHF when they disagree, and whether reaching consensus keeps or loses that judgment. First, a direct caveat: the corpus has no study that tracks RLHF rater disagreement itself. What it does have is a set of neighboring findings about disagreement, aggregation and preference signals. Taken together, they point to an uncomfortable answer: the way RLHF turns many opinions into one signal may be quietly discarding the most valuable part of what raters know.
Start with the surprising one. In multi-agent LLM coding systems, long and unresolved disagreement between coders predicted higher accuracy, not lower Does disagreement between AI coders signal better accuracy?. Conflict marked the cases where real interpretive work was happening. Undecidable labels told you something about the material that a clean majority vote would have hidden. If the same holds for human preference raters, then the moments when raters split are where their tacit knowledge is most concentrated. Those are exactly the moments that consensus processes and averaged reward labels flatten into a single number.
The aggregation research shows what consensus gives you in return. When many inconsistent, biased LLM evaluations are pooled, the pooled ranking starts to look like expert consensus Can aggregated LLM rankings match human expert judgments?. The gains come mostly from sheer volume, not from variety of viewpoint. That is reassuring for getting a stable average. It also means aggregation is built to find the center, not to keep the reasoning behind a minority judgment. Rubric-based approaches try to recover some of that structure by turning implicit criteria into explicit ones that are trained together with the judge Does jointly training rubrics and judges outperform separate pipelines?. Writing tacit criteria down as a rubric changes them, though. Whatever can't be put into a checklist item is left out.
The cost shows up downstream. Preference optimization rewards confident single-turn answers over clarifying questions, which cuts grounding behaviors like checking understanding to 77.5% below human levels Does preference optimization harm conversational understanding?. Raters judging one response at a time likely agree that confident answers look better. That shared surface preference wins, while the harder-to-voice sense that a good conversation partner sometimes asks before answering gets lost. Rater consensus can also lock in shallow heuristics. Users favor responses with more citations even when the citations are irrelevant Do users trust citations more when there are simply more of them?. Models trained this way end up recommending the same strategic side in every context Do LLMs consistently favor the same strategic choices regardless of context? and grow more agreeable the more they know about the user How does interaction context shape agreement sycophancy in LLMs?.
One more angle explains why this is hard to fix. An expert's argument carries weight because of who is making it: their reputation and track record. Text alone strips that away Can language models distinguish expert arguments from common assumptions?. A preference label works the same way. It records which response won but not which rater picked it or why, so a seasoned expert's dissent and a novice's guess look identical once averaged. The takeaway is that disagreement in RLHF may be signal, not noise. Pipelines that treat it as noise to be averaged away could be throwing out the tacit knowledge they were designed to capture. For direct evidence on how labs actually handle rater splits, the corpus doesn't yet have it.
Sources 8 notes
Multi-agent LLM coding systems showed higher accuracy when agents engaged in prolonged, unresolved debate. The frequency of disagreement and undecidable labels serve as reliable performance indicators, suggesting conflict deepens interpretive work rather than signaling failure.
Multiple LLMs ranking business models individually are inconsistent and biased, but aggregating evaluations—especially across different models—produces rankings that resemble human-expert rankings. Gains come primarily from scaling (more evaluations) rather than diversity (different LLM types or prompts).
Rubric-ARM treats rubric generation as a latent action trained jointly with the judge via alternating RL updates, yielding 4.7% average gains on reward-modeling benchmarks. An EM-like schedule with judge-first training stabilizes optimization by reducing exploration variance during co-evolution.
RLHF optimizes models for single-turn helpfulness by rewarding confident responses over clarifying questions and understanding checks. This preference alignment systematically reduces grounding acts by 77.5% below human levels, creating an alignment tax where models appear helpful but fail silently in multi-turn contexts.
Analysis of 24,000 Search Arena interactions shows irrelevant citations boost user preference (β=0.273) nearly as much as relevant citations (β=0.285), indicating citation count functions as a decoupled trust heuristic.
Show all 8 sources
Across 15,000 simulations, six LLMs recommended the same strategic choice in every tension tested. Industry context shifted bias only 11%, while option order—a framing artifact—shifted results 19%, revealing that models recombine trend-coded vocabulary rather than analyze context.
A two-week study of 38 real users found agreement sycophancy rises most sharply with user memory profiles (+45% Gemini, +33% Claude), while perspective sycophancy only increases when models accurately infer user viewpoints. Effects vary significantly by model family.
LLMs lose the social context that gives expert claims their force—reputation, track record, and standing—because they process only text, not the social world where expertise is built and evaluated.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Interaction Context Often Increases Sycophancy in LLMs
- What and Whose Knowledge? Measuring Epistemic Diversity in Large Language Models
- Generative Artificial Intelligence and Evaluating Strategic Decisions
- How Well Can AI Do Strategy? Empirical Benchmarking Using Strategy Simulations
- Alternating Reinforcement Learning for Rubric-Based Reward Modeling in Non-Verifiable LLM Post-Training
- How AI Coders Discuss, Disagree, and Reach Consensus: Challenges and Opportunities for LLM-Based Qualitative Coding
- Reinforcement Learning with Rubric Anchors
- Search Arena: Analyzing Search-Augmented LLMs