Does an AI agree more with one particular person than with the crowd — or is it really a consensus machine?
Do individual language models match particular human judges better than population averages?
This explores whether a single LLM lines up with one specific person's judgments better than it lines up with the average of a whole group.
This explores whether a single LLM lines up with one specific person's judgments better than it lines up with the average of a whole group. The corpus leans toward the opposite: off-the-shelf models behave like consensus machines. The clearest evidence is on social norms. GPT-4.5 judged social appropriateness across 555 scenarios more accurately than every individual human did Can AI learn social norms better than humans?. Any one person is a noisy sample of their community's view. The model acts like that community's center of gravity, so it hits the shared answer more reliably than any single person Can AI predict social norms better than humans?.
Averaging is also why agreement with a crowd is the easy thing to measure. In Chatbot Arena, 240K+ crowdsourced votes produce a ranking that agrees with expert raters, because pooling many judgments washes out individual quirks Can crowdsourced votes reliably rank language models?. A model that matches the pooled signal tells you little about whether it tracks any one voter. Aggregates are stable and individuals are not, so the two kinds of match are different tests.
The models also aren't very individual themselves. Across 106 LLMs and 625 value scenarios, the models cluster in a narrow region of value space while human respondents scatter widely Do large language models actually reflect human value diversity?. Across 70+ models, open-ended outputs converge on strikingly similar answers, an 'Artificial Hivemind' Do different AI models actually produce diverse outputs?. Low-resource cultures get represented through dominant-culture proxies inside the model's internal states Do LLMs represent low-resource cultures through dominant cultural proxies?. The social-norm models even share the same systematic errors Can AI learn social norms better than humans?. Swapping one model for another doesn't give you a different judge. They all sit near the same idealized average, and none of them is a particular person.
Matching an individual seems to take extra work. LLMs finetuned on psychology experiment data predicted human decisions better than traditional cognitive models, and their embeddings captured individual differences between people Can language models learn to model human decision making?. That points to individual fit coming from training on people's actual behavior, not from a stock model. The corpus has no direct head-to-head of one model against one named judge versus the group average, so that comparison is an open gap.
Sources 7 notes
GPT-4.5 outperformed every individual human at judging social appropriateness across 555 scenarios, challenging the theory that embodied cultural experience is necessary. However, all AI models share identical systematic errors on unwritten norms.
GPT-4.5 outperforms all individual humans at predicting social appropriateness, yet structurally cannot enter the community processes that establish and validate norms. This reveals a critical gap between pattern-matching and authentic participation in knowledge-making.
Chatbot Arena's 240K+ crowdsourced preference votes produce credible model rankings because the underlying questions are diverse and discriminating, and crowd judgments correlate with expert raters—validating human preference as a scalable evaluation signal.
Analysis of 106 LLMs across 625 scenarios shows they cluster in a concentrated region of value space while human respondents scatter widely. Models are poor surrogates for diverse populations despite exhibiting coherent value systems.
INFINITY-CHAT analyzed 70+ models across 26K open-ended queries and found an "Artificial Hivemind" effect: models independently generate strikingly similar or identical responses due to overlapping training data and alignment procedures, undermining the diversity benefits of model ensembles.
Show all 7 sources
Mechanistic interpretability analysis reveals that low-resource cultures like Ethiopia and Algeria are structurally represented through high-resource cultural proxies in internal model states, not just output. This architectural bias persists even when models can produce correct surface-level answers.
LLMs finetuned on psychology experiment data predict human behavior more accurately than theory-driven models in decision tasks, capture individual differences in their embeddings, and transfer learning across tasks without task-specific design.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- On Epistemic Diversity in Large Language Models
- AI Models Exceed Individual Human Accuracy in Predicting Everyday Social Norms
- Artificial Hivemind: The Open-Ended Homogeneity of Language Models (and Beyond)
- NoveltyBench: Evaluating Language Models for Humanlike Diversity
- Learning Pluralistic User Preferences through Reinforcement Learning Fine-tuned Summaries
- Humans learn to prefer trustworthy AI over human partners
- Large Language Models Reflect the Ideology of their Creators
- SOTOPIA: Interactive Evaluation for Social Intelligence in Language Agents