You can't train an AI to stop making things up — so how do you build systems that catch it before it acts?
How does hallucination in LLMs affect agent action reliability?
This explores what happens when an LLM's tendency to make things up moves from a chat window into an agent that takes real actions, and what the corpus says actually keeps those actions reliable.
This explores what happens when an LLM's tendency to make things up moves from a chat window into an agent that takes real actions, and what the corpus says actually keeps those actions reliable. The short version is that you can't train hallucination out of the model, so a reliable agent has to be built to catch it. One formal result proves that any computable LLM will hallucinate on infinitely many inputs, whatever its architecture, and that self-correction can't remove this limit Can any computable LLM truly avoid hallucinating?. A related finding says self-improvement is capped by the generation-verification gap: a model can't reliably check work it couldn't reliably produce What limits autonomous capability in large language models?. For an agent, this means a fabricated file path, API parameter, or 'task complete' message will eventually happen. The real question is whether anything stops it before it turns into an action.
The corpus raises a point that changes how you look for these errors. Some argue 'hallucination' is the wrong word, because the model produces accurate and inaccurate outputs through exactly the same statistical process Should we call LLM errors hallucinations or fabrications?. Nothing misfires when an agent invents a tool argument. It's doing what it always does. That's why calling it 'fabrication' points toward verification rather than grounding as the fix Does calling LLM errors hallucinations point us toward the wrong fixes?. It's also why the model's own confidence is a weak alarm. One retrieval system flags risk using how often entities appear together in the training data, and it catches fabrications even when the model is highly confident Can pretraining data statistics detect hallucinations better than model confidence?. Setting temperature to zero doesn't help either: you get the same answer every time, but it's still a single draw that may be wrong Does setting temperature to zero actually make LLM outputs reliable?. An agent that repeats the same mistake reliably isn't a reliable agent.
Acting in the world also gives agents a fix that pure chat lacks: the environment can answer back. ReAct alternates reasoning steps with real tool calls and observations, so each step gets checked against outside feedback before the next one builds on it. On knowledge-heavy and interactive tasks it beat pure chain-of-thought by 10-34 points Can interleaving reasoning with real-world feedback prevent hallucination?. The danger is error propagation. One made-up fact early in a long chain of actions spreads into everything after it. Frequent contact with the real world is what breaks that chain. The flip side is that when a setup removes that contact, failures get hidden. Simulated agents look socially competent when one model plays every role, then fail systematically once each agent has private information it has to work to learn Why do LLMs fail when simulating agents with private information?.
In agent systems, hallucination also has relatives that aren't about facts. Multi-agent setups show role flipping, empty 'flake' replies, infinite loops, and conversations drifting off topic, all traced to LLMs lacking a persistent representation of their goal and role Why do autonomous LLM agents fail in predictable ways?. These are made-up facts about the agent itself: who it is and what it was doing. That's why the strongest reliability argument in the corpus is architectural. Reliable agents move memory, skills, and interaction protocols out of the model and into a surrounding harness, so the model isn't rebuilding its state from scratch at every step Where does agent reliability actually come from?. AgentFly goes further, letting agents improve through stored case, subtask, and tool memories without changing any weights Can agents learn continuously from experience without updating weights?.
There is some progress at the model level. Reducing 'attention noise', meaning over-attention to irrelevant context, lowers hallucination rates and also makes in-context learning more stable Can fixing attention noise solve multiple LLM failures at once?. That lowers how often errors happen, but the inevitability result says it can't bring them to zero. The insight that ties all this together is that agent reliability isn't something the model has. It's something the system around the model provides, through outside checks, external memory, and verification triggers that don't depend on the model judging itself. The corpus is stronger on these principles than on measured hallucination rates in deployed agents. If you're after numbers on how often agents take fabricated actions in production, it doesn't have much of that.
Sources 12 notes
Three formal theorems prove that any computable LLM must hallucinate on infinitely many inputs, and internal mechanisms like self-correction cannot eliminate this mathematical constraint. External safeguards are therefore necessary, not optional.
Multi-agent deliberation produces specific failure modes (Degeneration-of-Thought, Silent Agreement), alignment at scale includes problematic self-valuation, and self-improvement is formally bounded by the generation-verification gap. Measurement error and conditional compliance hide the true capability ceiling.
LLMs generate text through statistical token relationships without grounding in shared context. Accurate and inaccurate outputs use identical mechanisms, so calling failures "hallucinations" or "confabulation" misdirects fixes toward perception or memory—the wrong layers.
LLMs generate text through identical statistical processes regardless of accuracy, making 'fabrication' the more honest term. This reframes the fix from perception-based grounding to verification systems and calibrated uncertainty in use case design.
QuCo-RAG uses entity co-occurrence patterns from training data to trigger retrieval, successfully flagging hallucination risk even when models are highly confident. This data-side approach catches the root cause (unseen combinations) rather than the symptom (low confidence).
Show all 12 sources
Fixed seeds and zero temperature replicate the same output repeatedly, but that output remains one draw from the model's probability distribution. McDonald's omega testing across 100 repetitions reveals that consistency does not equal reliability.
ReAct demonstrates that alternating verbal reasoning with external tool queries (Wikipedia API, environment interaction) prevents error propagation by injecting real-world feedback at each step. On knowledge-intensive and interactive tasks, this approach outperforms pure chain-of-thought and reinforcement learning by 10-34% absolute accuracy.
Research shows LLMs perform well when one model controls all interlocutors but fail systematically when agents possess private information. This reveals that apparent social competence relies on grounding work that models skip in omniscient settings.
Research identifies role flipping, flake replies, infinite loops, and conversation deviation as LLM-specific failures in multi-agent cooperation. These occur because LLMs lack persistent goal representation and stable role identity.
Research shows reliable LLM agents externalize three cognitive burdens—memory (state persistence), skills (procedural components), and protocols (structured interaction)—into a harness layer rather than relying on model scale alone. The harness unifies these externalities and eliminates the need for the model to solve the same problems repeatedly.
AgentFly formalizes agent learning as a Memory-augmented MDP with three memory modules (case, subtask, tool) that enable credit assignment and policy improvement entirely through memory operations. The approach achieved 87.88% on GAIA validation without modifying LLM parameters.
Cancelling attention noise through differential attention improves hallucination rates, makes in-context learning robust to example order, and reduces activation outliers—suggesting these separate failure modes share one root cause: over-attention to irrelevant context.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- A Comprehensive Survey of Hallucination Mitigation Techniques in Large Language Models
- Hallucination is Inevitable: An Innate Limitation of Large Language Models
- Chain-of-Verification Reduces Hallucination in Large Language Models
- A comprehensive taxonomy of hallucinations in Large Language Models
- The Model Says Walk: How Surface Heuristics Override Implicit Constraints in LLM Reasoning
- Query Rewriting for Retrieval-Augmented Large Language Models
- Detecting hallucinations in large language models using semantic entropy
- Useful Memories Become Faulty When Continuously Updated by LLMs