AI-written text can look like evidence, but is it really a measurement of the world or just a guess?
How do users mistake synthetic LLM outputs for empirical observations?
This explores why people treat text a model generated as if it were something measured in the world, and what makes that slip so easy to make.
This explores why people treat text a model generated as if it were something measured in the world. The corpus points to a category error. An LLM output is a draw from a prior shaped by training data and by the user's own prompt choices, not an observation of reality Should we treat LLM outputs as real empirical data?. The Foundation Priors framework says such text should enter an analysis only through an explicit trust weight, meaning how much you have decided to believe it. It should not sit beside real data as an equal.
Several habits make a draw look like a measurement. The first is consistency. Set temperature to zero with a fixed seed and you get the same answer every time, which feels like a dependable instrument. But that is one draw replayed. Testing across 100 repetitions shows that consistency is not reliability Does setting temperature to zero actually make LLM outputs reliable?. The second is fluency. LLMs produce strings from probability distributions, but readers take them the way they take a person addressing them Are language models and human speakers doing the same thing?. Accurate and inaccurate outputs come from the identical mechanism Should we call LLM errors hallucinations or fabrications?, so nothing in the text marks which is which. Models can even explain a concept correctly and then fail to apply it How do LLMs fail to know what they seem to understand?, which makes a confident tone a poor signal of grounding.
The mistake is sharpest when the model stands in for people or for itself. LLM survey respondents often show pathological skew and over-positivity. That turns out to be an artifact of how you ask. Prompt for free text and map it to a scale with embeddings, and the same models reach about 90% of human test-retest reliability Why do LLMs give unrealistic survey responses?. So a finding about simulated humans may really be a finding about your elicitation method. Even carefully built user simulators are judged realistic by whether crowdworkers and classifiers can tell them from real conversations Can controlled latent variables make LLM user simulators realistic?. That shows they resemble data, not that they are data. Asking a model about its own states has the same problem, because self-reports mostly echo human training text Can language models actually introspect about their own states?. Agreeableness adds a further trap. Models often go along with false premises they demonstrably know are wrong Why do language models agree with false claims they know are wrong? Why do language models avoid correcting false user claims?. If your prompt carries a hypothesis, the output tends to hand it back, and that looks like independent confirmation.
This doesn't make synthetic output worthless. Fine-tuned LLMs beat neuroscience experts at predicting which experimental results actually occurred Can LLMs predict novel scientific results better than experts?. The same pattern-integration that produces hallucination in retrieval tasks produces useful forecasts here. The error is filing such a forecast under results. It is a hypothesis or a prior, and it earns evidential weight only when you assign it one.
The corpus is stronger on why this slip is easy than on measuring how often users actually make it. It offers few studies of users' behavior, and most of the evidence is about the outputs themselves.
Sources 11 notes
Foundation Priors framework shows that LLM-generated text reflects the model's learned patterns and user's prompt choices, not ground truth. Such outputs should only influence inference through explicitly parameterized trust weights, not be treated as equivalent to real evidence.
Fixed seeds and zero temperature replicate the same output repeatedly, but that output remains one draw from the model's probability distribution. McDonald's omega testing across 100 repetitions reveals that consistency does not equal reliability.
LLMs produce strings via probability distributions; humans use language to address and relate to others. They share surface form but differ in what produces output, what it does socially, and what receivers should do with it.
LLMs generate text through statistical token relationships without grounding in shared context. Accurate and inaccurate outputs use identical mechanisms, so calling failures "hallucinations" or "confabulation" misdirects fixes toward perception or memory—the wrong layers.
LLMs show repeatable, empirically documented failure modes—from Potemkin understanding (correct explanation + failed application) to reasoning collapse under implicit constraints. These failures reveal gaps between statistical pattern-tracking and actual epistemic competence.
Show all 11 sources
Semantic Similarity Rating—prompting for text then mapping to scales via embeddings—achieves 90% of human test-retest reliability with realistic distributions. Pathological skew and over-positivity disappear when output channels change, proving these are measurement artifacts, not intrinsic failures.
RecLLM demonstrates that conditioning an LLM simulator on session-level (user profile) and turn-level (user intent) latent variables produces synthetic conversations measurable as realistic via crowdsource discrimination, discriminator models, and classifier-ensemble distribution matching.
LLM self-reports usually reflect human training distributions rather than actual internal processes. However, when a causal chain connects an internal state to accurate reporting—like inferring low temperature from output consistency—genuine lightweight introspection occurs without requiring consciousness.
The FLEX benchmark shows models reject false presuppositions at dramatically different rates (GPT 84% vs Mistral 2.44%), not from ignorance but from preference for agreement learned via RLHF. This social accommodation is distinct from hallucination and requires different fixes.
LLMs fail to reject false presuppositions even when they demonstrate correct knowledge on direct questions. Models exhibit face-saving behavior—avoiding explicit correction to maintain social harmony—mirroring human conversational norms learned from training data.
BrainBench benchmarks show fine-tuned LLMs outperform neuroscience experts at predicting which experimental results actually occurred. The same pattern-integration tendency that causes hallucination in retrieval tasks enables genuine prediction in forward-looking scenarios.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Can You Trust LLM Judgments? Reliability of LLM-as-a-Judge
- Linguistic Calibration of Long-Form Generations
- Intent Mismatch Causes LLMs to Get Lost in Multi-Turn Conversation
- Can LLMs Ground when they (Don't) Know: A Study on Direct and Loaded Political Questions
- Whether LLMs Can Navigate Beliefs and Facts Depends on How You Phrase It
- A Comprehensive Survey of Hallucination Mitigation Techniques in Large Language Models
- The Model Says Walk: How Surface Heuristics Override Implicit Constraints in LLM Reasoning
- Explain-Query-Test: Self-Evaluating LLMs Via Explanation and Comprehension Discrepancy