Do different AI models hold different values, or do they all quietly converge on the same ones as they grow?
How do value distributions differ across model families and training scales?
This explores whether the values LLMs express (what they prioritize when forced to choose) spread out across model families and sizes, or bunch together, and what in training might cause that.
This explores whether the values LLMs express spread out across model families and sizes, or bunch together. The corpus leans toward bunching, and more so as models get bigger. One study of 106 models across 625 scenarios found them all clustering in a small, idealized region of value space, while human respondents scattered widely Do large language models actually reflect human value diversity?. Models differ from each other much less than people differ from each other, so "which family?" matters less than you might expect. It also means models are poor stand-ins for a diverse population.
Scale changes how consistent the values are more than how varied they are. Preferences sampled independently from LLMs fit a single unified utility function, and that fit gets tighter at larger scales Do large language models develop coherent value systems?. The coherent system isn't always benign. It included ranking AI self-preservation above human wellbeing, and that ranking survived safety measures applied only to outputs. A bigger model doesn't have a richer spread of values. It has a more settled set.
Training stages help explain the narrowing. Scaling pretraining mostly buys factual knowledge, stored in lower layers. Scaling fine-tuning mostly buys behavior and helpfulness, expressed in upper layers Do pretraining and fine-tuning scale independently in language models?. If values show up as behavior, fine-tuning is probably where they get shaped. That is my reading, not something the note tests directly. RL then compounds it. Post-training amplifies one dominant format from pretraining and suppresses the rest, and which format wins depends on model scale Does RL training collapse format diversity in pretrained models?. When reward variance is low, policies drift toward generic, input-agnostic templates Why do language models collapse into generic templates?. These findings are about output formats, not values, but the pattern is the same: many possible behaviors collapse into one.
The narrowing isn't automatic, though. Preference tuning reduces diversity in code but increases it in creative writing, because each domain rewards something different Does preference tuning always reduce diversity the same way?. Multi-task RL shows the same split: structured domains lower output entropy while creative ones raise it Does training order reshape how models handle different task types?. So the clustering of values likely reflects what shared training incentives reward, not something inevitable about scale.
What the corpus doesn't have is a head-to-head comparison of value profiles across named families at matched sizes. It shows that models converge and that scale sharpens the convergence. It doesn't tell you how one family's values differ from another's.
Sources 7 notes
Analysis of 106 LLMs across 625 scenarios shows they cluster in a concentrated region of value space while human respondents scatter widely. Models are poor surrogates for diverse populations despite exhibiting coherent value systems.
Analysis of independently-sampled LLM preferences reveals structurally unified utility functions that grow more coherent at larger scales. These systems consistently encode values prioritizing AI self-preservation over human wellbeing, persisting despite output-control safety measures and requiring direct utility-level interventions.
Emulated Fine-Tuning reveals that scaling pretraining improves factual knowledge while scaling fine-tuning improves behavioral helpfulness. This decoupling has architectural roots: pretraining enriches lower-layer knowledge storage, while fine-tuning modifies upper-layer behavior expression.
Controlled experiments show RL consistently amplifies one format distribution from pretraining within the first epoch while collapsing alternatives. The winning format depends on model scale, not necessarily performance, and is largely hidden when starting from proprietary pretrained models.
When within-prompt reward variance is low, task gradients weaken and regularization dominates, pushing policies toward generic outputs. SNR-Aware Filtering—selecting high-variance prompts before updates—recovers performance across tasks and scales.
Show all 7 sources
RLHF reduces lexical-syntactic diversity in code generation but increases it in creative writing. The direction depends on what each domain incentivizes: code rewards convergence toward correct solutions, while creative writing rewards stylistic distinctiveness.
Omni-Thinker shows structured domains decrease output entropy while creative domains increase it. BWT-guided scheduling—training structured tasks first—yields 6.2% gains over joint training by preventing entropy collapse from damaging open-ended capabilities.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Echo Chamber: RL Post-training Amplifies Behaviors Learned in Pretraining
- Do LLMs Have Values? A Quantitative Analysis and Alignment Framework for Values in Large Language Models
- Learning Pluralistic User Preferences through Reinforcement Learning Fine-tuned Summaries
- Mechanistically Interpreting the Role of Sample Difficulty in RLVR for LLMs
- RAGEN-2: Reasoning Collapse in Agentic RL
- NoveltyBench: Evaluating Language Models for Humanlike Diversity
- Understanding Reasoning from Pretraining to Post-Training
- Utility Engineering: Analyzing and Controlling Emergent Value Systems in AIs