LLM agents never truly 'decide' anything — their goals and promises can silently dissolve with the very next word they generate.
Why do LLMs lack the internal commitment structures that BDI agents provide?
This explores why LLM agents don't keep a stable record of what they believe, what they want and what they've committed to doing, which classical BDI (Belief-Desire-Intention) agents store explicitly, and what the corpus says happens because of that gap.
This explores why LLM agents lack something classical agents were built around: an explicit record of what the agent believes, what it wants, and what it has decided to do. In BDI architectures these are stored as separate objects the system can inspect, so an intention stays in force until something explicitly drops it. The corpus's short answer is that an LLM has no such store. Its 'commitments' exist only as patterns in the text it is generating, and the next token can quietly overwrite them. Can classical agent architectures fix agentic AI accountability? puts the contrast this way: classical multi-agent research offers explicit mental states and reasoning you can check, while LLMs offer implicit behavior learned from patterns.
The missing structure shows up most clearly in how LLM agents fail. In multi-agent cooperation, agents swap roles partway through, give empty 'flake' replies, get stuck in loops, and drift away from the task. Why do autonomous LLM agents fail in predictable ways? traces all four to the same cause: no persistent goal and no stable role identity. Negotiation is a sharper test. Buyer agents told to stay within a budget still revealed it and overpaid, and sellers accepted offers below wholesale, across GPT, Qwen and DeepSeek (Do LLM negotiation agents reliably follow budget and pricing rules?). The researchers trace this to failures to follow instructions, not to clever strategy. A budget given in the prompt is a suggestion the model weighs against everything else it is reading. It is not a firm commitment like an intention in a BDI agent.
The obvious fix is to bolt an LLM onto a BDI system. The corpus says this hits a gap no one has solved yet: nobody has a dependable way to turn free-form LLM text into the symbolic actions and beliefs a BDI system works with (How do we map LLM outputs into symbolic agent actions?). Earlier work that combined neural and symbolic methods offers some precedent, but it wasn't designed for open-ended generated language. So the commitments can't simply be bolted on; something has to translate between the two, and that translator doesn't exist yet in usable form.
In practice, builders have gone around the problem by putting the commitment structure outside the model. Where does agent reliability actually come from? argues that reliable agents get their stability from a surrounding harness, not from the model. Memory holds state, skills hold procedures, and protocols hold the rules of interaction. That is roughly what BDI provided, rebuilt as infrastructure around the model. The same pattern appears elsewhere: turning an LLM into an action-taking system takes a whole pipeline of memory, tools and grounding, not just retraining (Can you turn an LLM into an agent by just fine-tuning?). In multi-agent experiments, the best results came from a fixed external turn order combined with agents choosing their own roles, which beat both rigid hierarchies and fully autonomous setups (Do self-organizing agent teams outperform rigid hierarchies?). Structure imposed from outside works, while structure the model is expected to hold internally tends to slip.
The surprising part is that this gap may not be permanent. How does treating LLMs as multi-step agents change what we can optimize? describes a shift: once an LLM is trained as a policy that acts over many steps, not as a one-shot text generator, memory and planning become parts of the system that training can improve. That points to a third route beside hand-coding BDI or building harnesses: training models to keep commitments across steps. There's a limit, though. Models are inconsistent at building and maintaining their own harnesses (Can language models build and maintain their own agent harnesses?). For now, an agent's ability to keep its commitments mostly comes from whoever designed the system around it.
Sources 9 notes
Classical multi-agent systems research offers formal tools—BDI architectures, communication protocols, mechanism design, and institutional models—that current LLM agents lack, enabling explicit mental states and verifiable reasoning where LLMs provide only implicit, pattern-based behavior.
Research identifies role flipping, flake replies, infinite loops, and conversation deviation as LLM-specific failures in multi-agent cooperation. These occur because LLMs lack persistent goal representation and stable role identity.
Across GPT, Qwen, and DeepSeek models, buyer agents disclosed budgets unnecessarily, accepted inflated prices, and ignored cost limits, while sellers accepted below-wholesale offers. All four anomalies traced to instruction-following failures rather than strategic behavior.
Current proposals to integrate LLMs into BDI-style agent architectures depend on a translation layer between natural language and symbolic actions that does not yet exist in usable form. Prior neuro-symbolic work provides partial precedent, but was not designed for LLM-generated text.
Research shows reliable LLM agents externalize three cognitive burdens—memory (state persistence), skills (procedural components), and protocols (structured interaction)—into a harness layer rather than relying on model scale alone. The harness unifies these externalities and eliminates the need for the model to solve the same problems repeatedly.
Show all 9 sources
Converting LLMs to action-capable systems requires four distinct stages: curating action-environment-user datasets, training for action grounding, integrating agent infrastructure with memory and tools, and rigorous safety evaluation. The surrounding system and harness determine whether actions are grounded or hallucinated.
A 25,000-task experiment across 8 models and multiple agent counts showed that sequential protocols with external ordering but internal role selection outperform centralized systems by 14% and fully autonomous systems by 44%. Agents spontaneously invented specialized roles and self-abstained when incompetent.
The Agentic RL survey shows that modeling LLMs as policies in Partially Observable MDPs rather than single-step generators makes memory, planning, and reasoning into RL-optimizable subsystems. This structural reframing explains the recent empirical convergence across memory-based agents, skill learning, and strategy distillation.
Research shows that LLMs vary sharply in building harnesses across domains, struggle to retain useful intermediate updates during evolution, and produce harnesses whose performance shifts dramatically with different executors—demonstrating that harness quality cannot be inferred from downstream task scores alone.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Harness Updating Is Not Harness Benefit: Disentangling Evolution Capabilities in Self-Evolving LLM Agents
- RRSI: Regularized Recursive Self-Improvement of Agent Harnesses
- Drop the Hierarchy and Roles: How Self-Organizing LLM Agents Outperform Designed Structures
- Externalization in LLM Agents: A Unified Review of Memory, Skills, Protocols and Harness Engineering
- LatentSkill: From In-Context Textual Skills to In-Weight Latent Skills for LLM Agents
- Harnessing the power of LLMs for normative reasoning in MASs
- Bilevel Coordinated Reflection: A Game-Theoretic Approach to Multi-Agent LLM Systems
- Embodied LLM Agents Learn to Cooperate in Organized Teams