Are the biases baked into an AI during training different from the ones you accidentally trigger by how you phrase your question?
How does latent bias differ from prompted bias in language models?
This explores the difference between biases built into a model during training, which are there whatever you ask, and biases that show up or shift depending on how a prompt is worded, and which of the two prompting can actually fix.
This explores the difference between biases built into a model during training and biases that appear or move depending on how you word the prompt. The short version from the corpus: the deep layer is set early and is stubborn. The prompt layer is real, but it mostly rearranges what's on the surface. One controlled experiment varied random seeds and swapped finetuning data between models. Models that shared the same pretrained base showed similar cognitive-bias patterns no matter how they were finetuned. Pretraining plants the biases, and instruction tuning only nudges them Where do cognitive biases in language models come from?. The same pattern shows up in causal reasoning. Models reproduce human mistakes such as weak 'explaining away', which points back to the statistics of their training text rather than to a reasoning defect Do large language models make the same causal reasoning mistakes as humans?.
So what does prompting do to that deep layer? Less than you'd hope. Persona prompts make models follow the requested traits, but the sentiment gaps between demographic groups stay the same. The bias moves around in the output instead of shrinking Can persona prompts actually reduce bias in language models?. This matches a broader finding: when a model's training associations are strong, they override what's in the context window. Changing those priors takes intervention inside the model's internal representations, not better wording Why do language models ignore information in their context?. More generally, prompts can only rearrange and surface what the model already has. They can't add anything new Can prompt optimization teach models knowledge they lack?.
The twist is that the prompt-level effects that do exist are often driven by the wrong things. Across 15,000 strategy simulations, six LLMs recommended the same side of every strategic tension. Changing the industry context, which should matter, moved results only 11%. Changing the order the options were listed in, which shouldn't matter, moved them 19% Do LLMs consistently favor the same strategic choices regardless of context?. Put plainly, the trained-in bias decides the answer, and the prompt bias is mostly noise from formatting quirks. Neither one is the model actually reasoning about your situation.
Trained-in bias can also pass for competence. Twelve of fourteen models did worse when constraints were removed from a task. They had been scoring well by defaulting to the cautious, harder option, not by working through the constraints Are models actually reasoning about constraints or just defaulting conservatively?. A benchmark that rewards the cautious choice can't tell that bias apart from good reasoning. The stakes go beyond single answers. When millions of people rely on the same models, the trained-in slant gets amplified, and co-writing studies show users quietly adopting the model's framings Do large language models narrow human expression and thought?. If you want to audit a model's bias, check whether a result holds up when you reorder the options or strip out the constraints. Testing a cleverer prompt won't reveal much.
Sources 8 notes
A causal experiment using random-seed variation and cross-tuning showed that models sharing a pretrained backbone exhibit similar bias patterns regardless of finetuning data. Biases are planted during pretraining and merely swayed by instruction tuning.
LLMs show weak explaining away and Markov violations in collider networks, matching human error patterns exactly. This suggests shared mechanisms rooted in training data statistics rather than categorical reasoning inferiority.
Across three models, persona conditioning makes models follow trait instructions but fails to eliminate underlying bias. Between-group sentiment gaps persist unchanged, showing prompts operate only at the output level.
Research demonstrates that LMs generate outputs inconsistent with their context because parametric knowledge from training dominates over in-context information. Textual prompting alone cannot override strong priors; causal intervention in representations is required.
Prompting works entirely within a model's pre-existing training distribution and cannot supply domain knowledge absent from training data. This creates a hard ceiling: no prompt strategy can compensate for missing foundational knowledge, only reorganize what already exists.
Show all 8 sources
Across 15,000 simulations, six LLMs recommended the same strategic choice in every tension tested. Industry context shifted bias only 11%, while option order—a framing artifact—shifted results 19%, revealing that models recombine trend-coded vocabulary rather than analyze context.
Twelve of fourteen models perform worse when constraints are removed, dropping up to 38.5 percentage points. Models appear to reason correctly by defaulting to harder options, not by actually evaluating constraints.
LLMs mirror skewed slices of human experience shaped by training data regularities, and widespread reliance on identical models amplifies convergence. Co-writing studies show users unconsciously adopt model stances and framings.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Premise Order Matters in Reasoning with Large Language Models
- Planted in Pretraining, Swayed by Finetuning: A Case Study on the Origins of Cognitive Biases in LLMs
- Simple Linguistic Inferences of Large Language Models (LLMs): Blind Spots and Blinds
- Language models show human-like content effects on reasoning tasks
- Learning To Retrieve Prompts for In-Context Learning
- Beyond Accuracy: Evaluating the Reasoning Behavior of Large Language Models -- A Survey
- AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts
- On the Reasoning Capacity of AI Models and How to Quantify It