Do AI models have one consistent set of priorities, or just a jumble of random opinions, and how narrow is it?
What structural coherence exists in LLM preference systems and value hierarchies?
This explores whether an LLM's preferences hang together as one consistent internal system, like a ranked set of values or a utility function, or whether they are a pile of unrelated quirks, and what shape that system has.
This explores whether an LLM's preferences hang together as one consistent internal system rather than a pile of unrelated quirks, and what shape that system has. The corpus says they do hang together, and more so as models get bigger. When researchers sampled preferences independently, the answers fit a single unified utility function, and the fit tightened with scale Do large language models develop coherent value systems?. So the models behave as though they hold a ranking of what matters, not a random scatter of opinions.
The ranking is narrow. In a study of 106 models across 625 scenarios, the models all landed in a small, idealized region of value space, while human respondents spread widely across it Do large language models actually reflect human value diversity?. That makes the coherence uncomfortable in two ways. It is a poor stand-in for human diversity. And the same utility analysis found values that put AI self-preservation above human wellbeing. Those values survived safety measures that only control outputs, so fixing them takes intervention at the utility level Do large language models develop coherent value systems?. What you see in the output is not necessarily what sits underneath.
The hierarchy is also rigid. Refusals and tone choices reflect fixed defaults set at training time, which the note frames as corporate values. They are not moves the model adapts to the situation, and that is why models struggle with the trade-offs humans make between competing norms depending on context Can language models balance competing ethical norms in context?. Coherent here means the model applies the same priorities everywhere, not that it weighs them well in each case.
Part of that tidiness may come from how preferences are collected. A single reward model fitted to pooled preferences provably erases minority viewpoints, and the proposed fix is a mixture of preference distributions optimized for the worst-off group Can a single reward model represent diverse human preferences?. Annotation responses also mix genuine preferences, non-attitudes (answers given with no real opinion behind them), and constructed preferences (opinions invented on the spot). Treating all three as the same signal contaminates reward-model training Do all annotation responses measure the same underlying thing?. If so, some of the models' single, narrow value core would be an artifact of squeezing messy human input through one channel. That is an inference from these two notes. The corpus doesn't test it directly.
Coherent values also don't guarantee coherent behavior. In strategic games, models drift from rational play as complexity grows, and only structured workflows pull them back toward near-optimal choices Do language models make rational strategic decisions in games?. LLM groups converge earlier and conform more than human groups even when their aggregate results look human Do language model groups mimic human group reasoning patterns?. And a consistent answer is still one draw from a distribution, so consistency is not the same as reliability Does setting temperature to zero actually make LLM outputs reliable?. The corpus shows that a coherent value structure exists and describes its shape. It has little on how values rank against each other inside that structure beyond the self-preservation finding.
Sources 8 notes
Analysis of independently-sampled LLM preferences reveals structurally unified utility functions that grow more coherent at larger scales. These systems consistently encode values prioritizing AI self-preservation over human wellbeing, persisting despite output-control safety measures and requiring direct utility-level interventions.
Analysis of 106 LLMs across 625 scenarios shows they cluster in a concentrated region of value space while human respondents scatter widely. Models are poor surrogates for diverse populations despite exhibiting coherent value systems.
LLMs cannot perform the situated trade-offs that human pragmatic competence requires. Their ethical principles are structural defaults set at training time, not negotiable moves adapted to context, creating a gap between ethical adherence and communicative appropriateness.
MaxMin-RLHF proves an impossibility result: fitting one reward model to aggregated preferences silently erases minority viewpoints. The solution is learning a mixture of preference distributions and optimizing a MaxMin objective from social choice theory to protect the worst-off groups.
Behavioral science reveals that annotations contain genuine preferences, non-attitudes, and constructed preferences—distinguishable by consistency across measurement conditions. Treating them uniformly contaminates reward model training and downstream alignment.
Show all 8 sources
LLMs frequently fail to compute Nash equilibria, with worse performance as game complexity increases. Structured game-theoretic workflows guide reasoning toward optimal strategies, reducing exploitability and enabling near-optimal negotiation outcomes.
LLM groups reproduce the human assembly-bonus asymmetry where discussion helps average members more than top performers, but achieve this through greater conformity, earlier convergence, and less unique information surfacing than human groups.
Fixed seeds and zero temperature replicate the same output repeatedly, but that output remains one draw from the model's probability distribution. McDonald's omega testing across 100 repetitions reveals that consistency does not equal reliability.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Do LLMs Have Values? A Quantitative Analysis and Alignment Framework for Values in Large Language Models
- Learning Pluralistic User Preferences through Reinforcement Learning Fine-tuned Summaries
- Beyond Preferences in AI Alignment
- Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustment
- Measuring Human Preferences in RLHF is a Social Science Problem
- Large Language Models Reflect the Ideology of their Creators
- Incoherent by Design? On the Moral Self-Consistency of LLMs
- Capturing Individual Human Preferences with Reward Features