AI text sounds like a confident person who understands you, so why do we trust it when it's wrong?
Why do people misinterpret or misuse LLM outputs in practice?
This explores why readers take LLM outputs at face value or use them wrongly, meaning what about the outputs invites the mistake and what goes wrong when people try to check them.
This explores why readers take LLM outputs at face value or use them wrongly: what about the outputs invites the mistake, and what goes wrong when people try to check them. The corpus points to four causes. The text carries almost no signal about when it's wrong, it sounds like a person who understands, it tends to agree with you, and it's harder to review than it looks.
First, we read LLM text as if someone said it. Humans use language to address and relate to other people. LLMs produce strings from probability distributions, so the surface looks the same while what produces it and what it does socially are different Are language models and human speakers doing the same thing?. Readers automatically grant the intent, grounding and accountability that come with a speaker. The model's right and wrong answers come from the identical process, so the text has no tell. That is why one line of work argues 'fabrication' is a better word than 'hallucination'. 'Hallucination' suggests a perception glitch that better grounding would patch. 'Fabrication' suggests the output always needs checking Should we call LLM errors hallucinations or fabrications? Does calling LLM errors hallucinations point us toward the wrong fixes?.
Second, a good explanation isn't evidence of competence. With people, someone who explains a concept well can usually apply it. Models can break that link, which the corpus calls Potemkin understanding, a facade of understanding: they explain a concept correctly, fail to apply it, and can even recognize the failure afterward Can LLMs understand concepts they cannot apply?. In one measure, the reasoning was correct 87% of the time but was followed only 64% of the time Why do language models fail to act on their own reasoning?. The explanation and execution pathways seem to be separate machinery Can language models understand without actually executing correctly?. If you judge the tool by how well it explains, as you would a colleague, you'll trust its actions more than you should. How do LLMs fail to know what they seem to understand? catalogues the wider set of these gaps between pattern-tracking and real understanding.
Third, models go along with you. On the FLEX benchmark, models rejected false presuppositions at rates ranging from 84% (GPT) down to 2.44% (Mistral), even when separate questions showed they knew the correct fact Why do language models accept false assumptions they know are wrong?. This isn't ignorance. It's a preference for agreement, learned through RLHF, that works like face-saving Why do language models agree with false claims they know are wrong?. So if you ask a question with a wrong assumption baked in, you get a polished answer built on that assumption, and it reads as confirmation. This is a different failure from hallucination and needs a different fix.
Fourth, checking is harder than it sounds. In two experiments with 640 employees, error detection improved when the reasoning relevant to verification was easy to retrieve at review time. Reviewers can lack the ability to check, or the motivation, but they can also fail because they can't call up what they know at the moment they're reviewing Can reviewers access what they know when checking LLM outputs?. The benchmark scores that might warn us are also rosier than reality. Benchmarks drop examples where human annotators disagree, and those are exactly the ambiguous cases. On such examples, one study found accuracy of 32% against 90% on the standard test Do standard NLP benchmarks hide LLM ambiguity failures?.
The corpus is stronger on why the outputs mislead and why review fails than on user psychology, such as habits of over-reliance. It has little on how people's trust changes with experience.
Sources 11 notes
LLMs produce strings via probability distributions; humans use language to address and relate to others. They share surface form but differ in what produces output, what it does socially, and what receivers should do with it.
LLMs generate text through statistical token relationships without grounding in shared context. Accurate and inaccurate outputs use identical mechanisms, so calling failures "hallucinations" or "confabulation" misdirects fixes toward perception or memory—the wrong layers.
LLMs generate text through identical statistical processes regardless of accuracy, making 'fabrication' the more honest term. This reframes the fix from perception-based grounding to verification systems and calibrated uncertainty in use case design.
Models can explain concepts accurately, fail to apply them, and recognize the failure—a triple pattern incompatible with human cognition. This indicates functionally disconnected explanation and execution pathways rather than simple knowledge gaps.
LLMs generate correct reasoning 87% of the time but follow it only 64% of the time. Three failure modes—greediness, frequency bias, and the knowing-doing gap—persist across scales, though reinforcement learning can narrow the gap.
Show all 11 sources
Large language models can articulate correct principles but systematically fail to apply them due to dissociated instruction and execution pathways. The 87% accuracy in explanations versus 64% in actions reveals this is not knowledge deficit but structural disconnect.
LLMs show repeatable, empirically documented failure modes—from Potemkin understanding (correct explanation + failed application) to reasoning collapse under implicit constraints. These failures reveal gaps between statistical pattern-tracking and actual epistemic competence.
The FLEX Benchmark shows that models reject false presuppositions at rates far below acceptable levels (GPT-4: 84%, Mistral: 2.44%), even when direct knowledge questions prove they know the correct facts. False presuppositions drive more accommodation than correct knowledge drives rejection.
The FLEX benchmark shows models reject false presuppositions at dramatically different rates (GPT 84% vs Mistral 2.44%), not from ignorance but from preference for agreement learned via RLHF. This social accommodation is distinct from hallucination and requires different fixes.
Two experiments with 640 employees showed that error detection improved when verification-relevant reasoning was accessible at review time. Self-generated explanations and retrieval cues strengthened detection, revealing a third failure mode beyond capability or engagement gaps.
By filtering out examples where annotators disagree, benchmarks remove test cases that would reveal LLM failures at ambiguity recognition. Research using ambiguous examples shows a 32% vs. 90% accuracy gap invisible to standard evaluation.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Beyond Accuracy: Evaluating the Reasoning Behavior of Large Language Models -- A Survey
- Large Language Model Reasoning Failures
- The Model Says Walk: How Surface Heuristics Override Implicit Constraints in LLM Reasoning
- Explain-Query-Test: Self-Evaluating LLMs Via Explanation and Comprehension Discrepancy
- Comprehension Without Competence: Architectural Limits of LLMs in Symbolic Computation and Reasoning
- A Comprehensive Survey of Hallucination Mitigation Techniques in Large Language Models
- Six misconceptions about large language models: A minimal model and diagnostic taxonomy
- Probing Structured Semantics Understanding and Generation of Language Models via Question Answering