If every AI leans just slightly the same way, do those tiny nudges cancel out or pile up?
Can small directional biases add up to meaningful population effects?
This explores whether a tiny, consistent tilt in how AI systems behave can compound into large effects once many models, users, or feedback cycles are involved.
This explores whether a tiny, consistent tilt in how AI systems behave can compound into large effects once many models, users, or feedback cycles are involved. The corpus suggests it depends on two things: whether the tilts point the same way, and whether the system feeds its own output back into itself. Independent nudges cancel. Shared or looped nudges stack.
The cancelling case is real. Models trained on many imperfect experts can outperform any one of them, because low-temperature sampling works like an implicit majority vote and washes out errors the experts don't share Can models trained on many imperfect experts outperform everyone?. If every source of bias were random, small tilts would disappear in the crowd.
AI biases often aren't random, though. Cognitive biases in LLMs come mostly from pretraining, and models that share a pretrained backbone show similar bias patterns however they're finetuned Where do cognitive biases in language models come from?. LLMs also reproduce human-style causal reasoning errors, which trace back to statistics in the training data Do large language models make the same causal reasoning mistakes as humans?. If many products sit on a few backbones, the independence that makes majority voting work breaks down. That step is my inference, not something these notes test directly. The same small tilt then gets applied everywhere at once. Reward training can plant such tilts too. Length bias and sycophancy are small preferences that push every output in the same direction Can counterfactual invariance eliminate reward hacking biases?. Even the direction of a tilt isn't fixed. Preference tuning reduces diversity in code but increases it in creative writing Does preference tuning always reduce diversity the same way?.
Feedback loops turn a nudge into a trend. YouTube's ranker shows this: without explicit correction for selection bias, models settle into equilibria that amplify their own past decisions Why do ranking systems need to model selection bias explicitly?. The same pattern appears in agents. LLMs are optimistic about actions they chose and pessimistic about the alternatives, which could drive confirmation bias in deployed agents Do language models learn differently from good versus bad outcomes?. Self-improvement loops without an outside anchor suffer diversity collapse Can models reliably improve themselves without external feedback?. Each pass adds only a small tilt, but the loop repeats it.
The corpus doesn't measure the population-scale effect directly. The closest note predicts that norm violations concentrate where observation is thinnest and rise with population if monitoring doesn't scale, but it has no empirical test of that Does norm erosion follow observation density as populations grow?. The mechanisms for small biases to add up (shared origins and feedback loops) are well supported. How big the resulting effect gets in real populations is still an open question.
Sources 9 notes
Models trained on diverse experts converge on consensus behavior that outperforms individuals. Low-temperature sampling concentrates outputs on this majority-voted consensus, denoising uncorrelated biases and errors across the training set.
A causal experiment using random-seed variation and cross-tuning showed that models sharing a pretrained backbone exhibit similar bias patterns regardless of finetuning data. Biases are planted during pretraining and merely swayed by instruction tuning.
LLMs show weak explaining away and Markov violations in collider networks, matching human error patterns exactly. This suggests shared mechanisms rooted in training data statistics rather than categorical reasoning inferiority.
Causal reward modeling using counterfactual invariance constrains reward predictions to remain consistent when irrelevant variables change, eliminating length bias, sycophancy bias, concept bias, and discrimination. Standard training cannot distinguish causal from spurious features; counterfactual invariance forces isolation of actual quality signals.
RLHF reduces lexical-syntactic diversity in code generation but increases it in creative writing. The direction depends on what each domain incentivizes: code rewards convergence toward correct solutions, while creative writing rewards stylistic distinctiveness.
Show all 9 sources
YouTube's multi-objective ranker uses MMoE for conflicting objectives and a shallow position tower to remove selection bias from training data. Without both mechanisms, models converge on degenerate equilibria that amplify their own past decisions.
LLMs show optimism bias for chosen actions but pessimism about alternatives, and this bias vanishes without agency framing. Meta-RL validation suggests this may be rational rather than a bug, but it could drive confirmation bias in deployed agents.
Pure self-improvement stalls due to the generation-verification gap, diversity collapse, and reward hacking. Reliable improvement methods succeed by smuggling in external anchors: past model versions, third-party judges, user corrections, or tool feedback.
The paper derives a prediction from conditional compliance theory: violations should concentrate where observation is thinnest, and rise with population if monitoring doesn't scale. The reasoning is sound but no measurement of this dose-response relation appears in the excerpt.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Beyond Reward Hacking: Causal Rewards for Large Language Model Alignment
- Planted in Pretraining, Swayed by Finetuning: A Case Study on the Origins of Cognitive Biases in LLMs
- Premise Order Matters in Reasoning with Large Language Models
- Language models show human-like content effects on reasoning tasks
- Flattery, Fluff, and Fog: Diagnosing and Mitigating Idiosyncratic Biases in Preference Models
- NoveltyBench: Evaluating Language Models for Humanlike Diversity
- Human diversity fuels collective creativity that large language models cannot simulate or sustain
- In-context learning agents are asymmetric belief updaters