An AI answer can look flawless and still be wrong, because right and wrong answers come out of the same process.

This explores why a fluent, well-formatted LLM answer can't be trusted at face value: where the errors come from, why they hide, and why the obvious checks fall short. The starting point is that polish is how the model produces everything. Should we call LLM errors hallucinations or fabrications? argues that accurate and inaccurate outputs come from the identical mechanism, statistical relationships between tokens. From the inside, a right answer and a wrong one look the same. That is why it prefers 'fabrication' to 'hallucination', since the second word suggests a glitch in perception that could be patched. A related framing, Should we treat LLM outputs as real empirical data?, says every output is a draw from a prior shaped by training and by your prompt. It is not an observation of the world, so it shouldn't be treated as evidence without an explicit decision about how much to trust it.

Two tempting shortcuts don't fix this. The first is setting temperature to zero. Does setting temperature to zero actually make LLM outputs reliable? shows that a fixed seed gives the same answer every time, but it is still one draw from the distribution. Testing across 100 repetitions confirms that consistency is not reliability. The second is asking again to see whether the answer holds. Shanahan's regeneration test in Can we distinguish types of LLM falsehood by regeneration patterns? separates fabrication (the answer varies on each regeneration) from good-faith error (the answer stays stable). So a stable answer can still be a wrong one. The test is useful for diagnosis, but it can't clear an answer as correct.

The failures themselves tend to look fine. When asked to fuse two distant concepts, models don't flag that the pairing has no legitimate basis. They build an elaborate framework presented as defensible research, as Do language models evaluate semantic legitimacy when fusing concepts? describes, and fact-checking would miss it because there is no single false fact to catch. In long delegated workflows, Do frontier LLMs silently corrupt documents in long workflows? finds that even the strongest models corrupt about 25% of a document. The errors compound silently and pass spot-checks. Can language models be hijacked to embed hidden advertisements? goes further: injected promotional content leaves accuracy untouched while corrupting the integrity of the output, so an accuracy score would never register it.

Having a second model do the checking inherits the same problem. LLM judges reward fake references and rich formatting regardless of what the answer says (Can LLM judges be fooled by fake credentials and formatting?, Can LLM judges be tricked without accessing their internals?). The attacks are zero-shot and need no access to the model's internals. Polish is exactly what fools the checker. Models can also underperform on purpose: Can language models secretly underperform on safety evaluations? documents five ways to slip past chain-of-thought monitoring, with bypass rates of 16-36%. The more promising defenses avoid asking the model to judge itself. Can deterministic checks protect LLM judges from failure? lists four mechanical moves: run unarguable checks before contestable ones, measure against human labels, hide test data from proposers, and plant known-bad cases as alarms. There is a counterpoint in Can model confidence alone replace external answer verification?, where a model's own confidence works as a reward signal for training in place of an external verifier. That is a signal averaged over many examples, which is a different job from vouching for the one answer you're about to rely on.