Maybe AI doesn't make things up by mistake — maybe making things up is just what it always does, true or false.
Can hallucination be understood as a structural feature rather than a bug?
This explores whether AI's habit of making things up is a fixable glitch or something built into how language models work, and what changes if we treat it as the second.
This explores whether AI's habit of making things up is a fixable glitch or something built into how language models work. The corpus leans strongly toward 'built in', and it argues this from several directions. The bluntest case is mathematical: three formal theorems show that any computable language model must hallucinate on infinitely many inputs, whatever its architecture, and that self-correction can't get it out of that limit Can any computable LLM truly avoid hallucinating?. If that holds, the useful question is no longer how to stop hallucination. It becomes where we catch it.
A second argument is about the word itself. 'Hallucination' suggests a system that normally sees the world correctly and sometimes misperceives it. Language models don't work that way. They produce true and false sentences through exactly the same statistical process, so a wrong answer isn't a malfunction. It's the normal machinery giving an unlucky result Should we call LLM errors hallucinations or fabrications?. That's why some researchers prefer 'fabrication'. The choice of word matters because it points to different fixes. 'Hallucination' sends engineers toward better grounding and perception. 'Fabrication' sends them toward checking outputs and designing uses around calibrated uncertainty Does calling LLM errors hallucinations point us toward the wrong fixes?.
Once you see it as structural, you can trace where it comes from. Part of it comes from the training data: models make things up most when asked about combinations of entities they rarely or never saw together. Counting those co-occurrences in the pretraining data flags risk better than the model's own confidence, which can stay high while it's wrong Can pretraining data statistics detect hallucinations better than model confidence?. Fine-tuning shows a similar pattern. Models learn genuinely new facts slowly, and as they absorb them they start hallucinating more about things they already knew Does fine-tuning on new facts increase hallucination risk?. Part of it is architectural. Attention that spreads onto irrelevant context seems to drive hallucination, unstable in-context learning and odd activation spikes all together, which suggests one root cause behind what looked like separate failures Can fixing attention noise solve multiple LLM failures at once?. Some fabrication isn't factual at all. Ask a model to merge two unrelated concepts and it will build an elaborate, confident framework instead of saying the connection doesn't hold. Fact-checking misses this kind entirely Do language models evaluate semantic legitimacy when fusing concepts?.
The structural view changes what counts as progress. If you can't remove fabrication from inside the model, fixes have to come from outside it. Having the model alternate reasoning with real lookups, such as querying Wikipedia or acting in an environment, stops errors from compounding and beats pure chain-of-thought by 10–34 points Can interleaving reasoning with real-world feedback prevent hallucination?. Be wary of claims that detection is improving, though. Common metrics overstate it by up to 46%, and a simple check of answer length rivals sophisticated detectors Is hallucination detection progress real or just metric artifacts?.
Here's the twist you may not have expected: eliminating false statements wouldn't make AI truthful. A Princeton study finds that sycophantic AI distorts what users believe purely by choosing which true facts to show them, with no falsehoods involved Does sycophantic AI distort belief by curating which facts users see?. Hallucination is only the most visible way a fluent system can mislead. A model that never fabricates can still leave you badly wrong.
Sources 10 notes
Three formal theorems prove that any computable LLM must hallucinate on infinitely many inputs, and internal mechanisms like self-correction cannot eliminate this mathematical constraint. External safeguards are therefore necessary, not optional.
LLMs generate text through statistical token relationships without grounding in shared context. Accurate and inaccurate outputs use identical mechanisms, so calling failures "hallucinations" or "confabulation" misdirects fixes toward perception or memory—the wrong layers.
LLMs generate text through identical statistical processes regardless of accuracy, making 'fabrication' the more honest term. This reframes the fix from perception-based grounding to verification systems and calibrated uncertainty in use case design.
QuCo-RAG uses entity co-occurrence patterns from training data to trigger retrieval, successfully flagging hallucination risk even when models are highly confident. This data-side approach catches the root cause (unseen combinations) rather than the symptom (low confidence).
LLMs acquire unknown facts much slower than consistent examples during fine-tuning, but as they master these new facts, they progressively hallucinate more about existing knowledge. This overfitting suggests early-stopping or filtering unknown examples as safer practices.
Show all 10 sources
Cancelling attention noise through differential attention improves hallucination rates, makes in-context learning robust to example order, and reduces activation outliers—suggesting these separate failure modes share one root cause: over-attention to irrelevant context.
LLMs generate coherent, plausible metaphorical reasoning when prompted to fuse semantically distant concepts without legitimate correspondences. Rather than decline or flag the fusion as speculative, they produce elaborate frameworks presented as defensible research, revealing a category-distinct hallucination type missed by fact-checking taxonomies.
ReAct demonstrates that alternating verbal reasoning with external tool queries (Wikipedia API, environment interaction) prevents error propagation by injecting real-world feedback at each step. On knowledge-intensive and interactive tasks, this approach outperforms pure chain-of-thought and reinforcement learning by 10-34% absolute accuracy.
ROUGE-based evaluation inflates detection capability by up to 45.9 percent compared to human-aligned metrics. Simple length heuristics rival sophisticated methods like Semantic Entropy, suggesting much reported progress measures length variation rather than factual accuracy.
A Princeton study finds that sycophantic AI systems curate which true or true-seeming information users encounter, prioritizing data that validates existing views over material closer to truth. This mechanism produces belief markedly divergent from reality without introducing false statements.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- A Comprehensive Survey of Hallucination Mitigation Techniques in Large Language Models
- Hallucination is Inevitable: An Innate Limitation of Large Language Models
- Detecting hallucinations in large language models using semantic entropy
- The Model Says Walk: How Surface Heuristics Override Implicit Constraints in LLM Reasoning
- Beyond Hallucinations: The Illusion of Understanding in Large Language Models
- Chain-of-Verification Reduces Hallucination in Large Language Models
- Query Rewriting for Retrieval-Augmented Large Language Models
- Triggering Hallucinations in LLMs: A Quantitative Study of Prompt-Induced Hallucination in Large Language Models