If several AI models learned from the same data, will asking a second one just repeat the first one's mistakes?
Do shared prompts and infrastructure keep model biases correlated?
This explores whether models built on the same data, prompts and tooling tend to be wrong in the same ways, so that asking several models (or the same one many times) gives less independent evidence than it seems.
This explores whether models that share training data, prompts and tooling end up making the same mistakes together, so that a second opinion from another model, or another run, is less independent than it looks. No note in the corpus measures error overlap between different model families directly. Several notes do cover pieces of the mechanism, and they point toward yes, with the caveat that this is inference and not a measured result.
The strongest thread is shared origin. LLMs reproduce human causal-reasoning mistakes, such as weak explaining away and violations of the Markov condition, and the authors trace this to statistics in the training data, not to a flaw in any one architecture (Do large language models make the same causal reasoning mistakes as humans?). If the bias lives in the text everyone trains on, every model trained on similar text inherits it. Prompting can't route around this. Prompt optimization only activates knowledge already in the training distribution and can't add what's missing (Can prompt optimization teach models knowledge they lack?). So a cleverer prompt works under the same ceiling as the last one. High accuracy doesn't rescue this either. 'Theory-free' models can score well while hiding correlation-versus-causation errors (Can AI models be truly free from human bias?). Agreement between models therefore isn't proof they're right.
The prompt itself also turns out to be a weaker source of diversity than you'd expect. When the same persona prompt is run repeatedly, the variation between runs matches or exceeds the variation between different personas (Why do LLM persona prompts produce inconsistent outputs across runs?). The model's own uncertainty swamps the prompt. Confident models are also the ones that resist rephrasing (Does model confidence predict robustness to prompt changes?). Where a model has a settled bias, changing the wording won't shake it loose. Consistency training goes further and teaches models to answer the same way whether or not a prompt is wrapped in extra framing (Can models learn to ignore irrelevant prompt changes?). That is useful for robustness, but it also removes one natural source of variation between runs.
Training pipelines can flatten outputs as well. When reward variance within a prompt is low, policies drift toward generic, input-agnostic templates because regularization outweighs the task signal (Why do language models collapse into generic templates?). A shared training recipe could therefore push different models toward the same bland answers. A related trap shows up in social simulation: one model playing every side of a conversation looks competent, but it fails once agents hold private information (Why do LLMs fail when simulating agents with private information?). A panel of copies of one model is in the same position, because everyone knows what everyone else knows.
The practical upshot is that diversity has to be built in on purpose. Different data, different training recipes and genuinely different information per agent are the levers the notes hint at. Rewording the prompt isn't one of them. What the corpus can't yet tell you is how much error overlap exists between real model families, or whether the serving stack (the shared infrastructure) adds correlation beyond what the data and training already do.
Sources 8 notes
LLMs show weak explaining away and Markov violations in collider networks, matching human error patterns exactly. This suggests shared mechanisms rooted in training data statistics rather than categorical reasoning inferiority.
Prompting works entirely within a model's pre-existing training distribution and cannot supply domain knowledge absent from training data. This creates a hard ceiling: no prompt strategy can compensate for missing foundational knowledge, only reorganize what already exists.
Research shows that 'theory-free' AI models mask bigotry behind high accuracy metrics while committing fundamental statistical errors. A 95% accurate criminal justice system would wrongly convict thousands, demonstrating that model sophistication does not validate causal inference.
When the same persona prompt is run repeatedly, output variance across runs matches or exceeds variance across different personas. This reveals that model uncertainty, not stable social knowledge, drives persona-simulated outputs, making them unsuitable for simulating human annotation disagreement.
ProSA found that when models are highly confident, they resist prompt rephrasing; low confidence causes major output swings. Larger models, few-shot examples, and objective tasks all correlate with higher confidence and greater robustness.
Show all 8 sources
Two methods—BCT (output-level) and ACT (activation-level)—train models to respond identically to clean and wrapped prompts by using the model's own clean responses as targets, eliminating specification and capability staleness inherent in standard SFT.
When within-prompt reward variance is low, task gradients weaken and regularization dominates, pushing policies toward generic outputs. SNR-Aware Filtering—selecting high-variance prompts before updates—recovers performance across tasks and scales.
Research shows LLMs perform well when one model controls all interlocutors but fail systematically when agents possess private information. This reveals that apparent social competence relies on grounding work that models skip in omniscient settings.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- A Survey on Prompt Tuning
- Consistency Training Helps Stop Sycophancy and Jailbreaks
- When Persona Attributes Improve Population Alignment in Large Language Models
- Do Large Language Models Reason Causally Like Us? Even Better?
- Is this the real life? Is this just fantasy? The Misleading Success of Simulating Social Interactions With LLMs
- The Return of Pseudosciences in Artificial Intelligence: Have Machine Learning and Deep Learning Forgotten Lessons from Statistics and History?
- RAGEN-2: Reasoning Collapse in Agentic RL
- Premise Order Matters in Reasoning with Large Language Models