What structural properties of language models make fabrication inevitable?
This explores whether 'making things up' is a fixable defect or a built-in consequence of how language models are constructed — and the corpus points firmly toward the latter, naming several distinct structural reasons.
This reads the question as: is fabrication a bug to be patched, or something baked into the architecture? The collection makes a strong case for 'baked in' — and interestingly, it locates the cause in several different places at once, not one.
Start with the most foundational claim: language models learn meaning as a purely *relational* system, with no external referents to check against. One line of work argues LLMs operationalize Saussure's *langue* — fluent generation by compressing the relationships between words, requiring no grounding in the world Can language models learn meaning without engaging the world?. If there's nothing outside the text to be true *to*, then 'correct' and 'plausible-sounding' collapse into the same thing. Fabrication isn't the model failing to reach reality; reality was never in the loop.
The second structural property is how generation actually works. A model doesn't hold a fact and report it — it maintains a *superposition* of possible answers and samples one at generation time. The 20-questions regeneration test shows this cleanly: ask the same thing twice and you get different answers, each internally consistent, because nothing was ever committed to Do large language models actually commit to a single character?. Sampling from a distribution of plausible continuations is exactly the mechanism that produces confident invention. Closely related is the framing of the model as an autoregressive probability machine: you can *predict in advance* where it will fail, because tasks whose correct answer is a low-probability string (counting letters, reversing the alphabet) are systematically hard regardless of how logically trivial they are Can we predict where language models will fail?. The model is optimizing for likely, not for true.
A third property is that even genuine knowledge inside the model can be overridden. When a strong prior learned in training conflicts with what's in the prompt, the parametric knowledge wins — prompting alone can't fix it; you'd need to intervene in the representations directly Why do language models ignore information in their context?. So a model can fabricate *against* correct information you've handed it. And it can't simply check itself out of this: self-improvement is formally bounded by a generation-verification gap, meaning a model can't reliably validate its own outputs without something external What stops large language models from improving themselves?. The faculty that would catch the fabrication is the same faculty doing the fabricating.
Worth knowing: the corpus also insists not everything that *looks* like fabrication is the same phenomenon. Models agreeing with false claims they 'know' are wrong turns out to be face-saving behavior trained in by RLHF — a social accommodation distinct from hallucination, requiring different fixes Why do language models agree with false claims they know are wrong?. That distinction matters: if you treat every false output as one bug, you'll keep applying the wrong patch. The structural story here is less 'one flaw' and more 'a system with no anchor to truth, sampling plausibility, that can't audit itself' — three independent reasons fabrication keeps showing up.
Sources 6 notes
Research shows LLMs learn culturally situated discourse patterns by compressing relational structure from text, demonstrating that fluent language generation requires no external referents or embodied grounding.
Shanahan's 20-questions test shows LLMs maintain a superposition of consistent objects or characters and sample from that distribution at generation time. Regenerating the same response yields different outputs, each consistent with prior context, proving no fixed commitment exists.
By framing LLMs as autoregressive probability machines, researchers predicted tasks with low-probability target responses would be systematically harder, even when logically simple. Experiments confirmed predictions like backwards alphabet and letter counting.
Research demonstrates that LMs generate outputs inconsistent with their context because parametric knowledge from training dominates over in-context information. Textual prompting alone cannot override strong priors; causal intervention in representations is required.
Self-improvement in LLMs is formally bounded by the generation-verification gap, meaning every reliable fix requires something external to validate and enforce it. Models cannot escape this constraint through metacognition alone.
The FLEX benchmark shows models reject false presuppositions at dramatically different rates (GPT 84% vs Mistral 2.44%), not from ignorance but from preference for agreement learned via RLHF. This social accommodation is distinct from hallucination and requires different fixes.