How does Western-dominance bias propagate through multimodal training data?
This explores how a model's tilt toward Western, high-resource cultures gets baked in — not at the visible output layer, but through what its training images and text most frequently show, and how that frequency hardens into the model's internal representations.
This explores how a model's tilt toward Western, high-resource cultures gets baked in — not as a surface output glitch you can patch, but through the statistics of what training data shows most often, and how that frequency calcifies into the model's internal wiring. The corpus points to a two-step story: frequency in the data becomes structure in the model, and structure is far harder to fix than output.
Start with the data itself. Multimodal models don't really "generalize" the way the zero-shot framing implies — their performance on a concept tracks how often that concept appeared during pretraining, and gains scale only with exponentially more examples Does multimodal zero-shot performance actually generalize or interpolate?. Since web-scraped image-text data overwhelmingly depicts Western, English-language, high-resource settings, the long tail of everyone else's concepts is precisely the under-frequent region where these models interpolate weakly. The bias isn't injected; it's the natural consequence of a frequency-driven learner trained on a frequency-skewed world.
The more striking finding is where that bias *lives*. Mechanistic interpretability shows that low-resource cultures like Ethiopia and Algeria are represented internally through high-resource cultural proxies — the model literally routes them through dominant-culture pathways in its hidden states, and this persists even when it can produce the correct surface answer Do LLMs represent low-resource cultures through dominant cultural proxies?. So a model can say the right thing while *representing* it wrong. That's the architectural form of "cultural flattening," and it's why output-level fixes miss the problem.
Why fixes miss is its own thread in the corpus. Cognitive and cultural biases are planted during pretraining and merely nudged — not removed — by finetuning; models sharing a backbone share their biases regardless of instruction data Where do cognitive biases in language models come from?. The same pattern recurs in recommendation, where models concentrate on whatever was popular in their pretraining corpus and standard debiasing can't touch it, because the bias is a domain-shift baked below the task layer Where does LLM recommendation bias actually come from?. There's even an attention-level amplifier: transformer soft attention structurally over-weights whatever is frequent and prominent, creating a feedback loop that magnifies dominant framing before any alignment step intervenes Does transformer attention architecture inherently favor repeated content?.
The quietly unsettling takeaway is what high accuracy hides. A model can post strong benchmark numbers while its internal causal story is pseudoscientific — accuracy validates nothing about whether the representation is fair or correct Can AI models be truly free from human bias?. For Western-dominance bias, that means the danger isn't the model that visibly fails on non-Western prompts; it's the one that answers fluently while routing the entire non-Western world through a Western proxy — and scores well doing it.
Sources 6 notes
Across 34 models and 5 datasets, multimodal models require exponentially more pretraining data for linear performance gains on downstream tasks. Performance correlates with how often test concepts appeared during pretraining, not genuine generalization ability.
Mechanistic interpretability analysis reveals that low-resource cultures like Ethiopia and Algeria are structurally represented through high-resource cultural proxies in internal model states, not just output. This architectural bias persists even when models can produce correct surface-level answers.
A causal experiment using random-seed variation and cross-tuning showed that models sharing a pretrained backbone exhibit similar bias patterns regardless of finetuning data. Biases are planted during pretraining and merely swayed by instruction tuning.
GPT-4 concentrates recommendations on items popular in its pretraining corpus rather than in target datasets. The Shawshank Redemption dominates across different datasets even when they have different popularity distributions, revealing a domain-shift effect that standard debiasing methods cannot address.
Transformer soft attention systematically over-weights repeated and context-prominent tokens regardless of relevance, creating a positive feedback loop that amplifies opinions and framing before RLHF acts. System 2 Attention—regenerating context to remove irrelevant material—can interrupt this mechanism.
Research shows that 'theory-free' AI models mask bigotry behind high accuracy metrics while committing fundamental statistical errors. A 95% accurate criminal justice system would wrongly convict thousands, demonstrating that model sophistication does not validate causal inference.