Should AI agents that negotiate in plain English get a stricter, machine-checkable rulebook underneath so promises can't be misread?
Do natural language protocols require formal semantics to govern agentic AI?
This explores whether AI agents that coordinate and delegate in plain natural language need something more rigorous underneath, like precise, machine-checkable rules for what a request or a promise means, to behave safely and accountably.
This explores whether agents that talk to each other and to us in ordinary language need a formal layer underneath that pins down what a request, a promise, or a permission actually means. The corpus leans toward yes, but with a twist. The useful formal layer is rarely a replacement for natural language. More often it's scaffolding that sits beside it, defining scope, commitments and points where the agent should stop and check.
The sharpest case comes from an older research tradition that LLM agents mostly skipped. Before LLMs, multi-agent systems researchers built communication languages like KQML and FIPA-ACL. Each message type in these languages, such as 'request,' 'inform' or 'promise,' had a precisely defined meaning, so both sides knew exactly what had been asked and what had been agreed. Natural language has no such guarantee. An agent told to look something up might decide on its own to contact a third party, overstepping what it was authorized to do (Do LLM agents misinterpret delegated requests more than formal protocols?). The broader argument is that this tradition, which also includes explicit belief-desire-intention models of agents, mechanism design and institutional rules, already offers the accountability tools LLM agents lack. Those older agents had mental states you could inspect. LLMs offer only implicit, pattern-based behavior (Can classical agent architectures fix agentic AI accountability?).
There's also a philosophical reason the meaning can't simply live in the words. One line of work argues that AI output carries the surface markers of real speech without the underlying event of someone actually addressing you. The human supplies the missing intent (Does AI generate genuine utterances or just text patterns?). A related argument says current AI systems can't genuinely assert anything, because they lack either a real descriptive function or the social accountability that makes an assertion binding (Can AI systems genuinely assert claims about the world?). If an agent's 'I'll do X' isn't a commitment in any meaningful sense, then the commitment has to be written down somewhere outside the model's prose. That is exactly what formal semantics do.
What's interesting is how much of this is coming back in lighter forms. OpenAI's Model Spec proposes a structured 'scope of autonomy' record, essentially a permissions slip listing the tools an agent may use, its cost limits, banned actions and when the task ends. It borrows the least-privilege principle from software security rather than a full speech-act theory (How can autonomous agents act safely within defined boundaries?). Conversation analysis offers a formal account of when an agent should pause and ask a clarifying question instead of silently chaining tools and drifting from what you wanted (When should AI agents ask users instead of just searching?). Formal argumentation can turn an AI's reasoning into a map of claims and counterclaims, so you can point to the exact premise you reject (Can formal argumentation make AI decisions truly contestable?). And one design proposal adds modules to LLM agents for discovering social norms, reasoning about them and following them. It hasn't been built or tested yet (Can LLMs discover and apply social norms dynamically in open systems?).
The surprising detour is that some researchers want to skip language altogether. One approach lets agents share latent 'thoughts' pulled straight from their internal states, with mathematical guarantees about what's shared and what stays private. That makes it possible to spot disagreements between agents before they ever show up in words (Can agents share thoughts directly without using language?). So the real question may not be 'natural language or formal rules.' It may be which layer of the agent you make precise. That could be the messages, the permissions, the decision points, or the agent's own internal representations. The corpus doesn't yet have head-to-head evidence on which of these matters most in practice.
Sources 9 notes
KQML and FIPA-ACL use formal speech-act semantics to guarantee shared understanding of requests and commitments, while LLM agents relying on natural language risk overstepping scope—such as querying third parties when only direct queries were authorized.
Classical multi-agent systems research offers formal tools—BDI architectures, communication protocols, mechanism design, and institutional models—that current LLM agents lack, enabling explicit mental states and verifiable reasoning where LLMs provide only implicit, pattern-based behavior.
AI output carries communicative markers inherited from training data but lacks the event structure that produces actual utterances. Users supply the missing orientation through interpretive labor, creating a pseudo-event with structure only on the human side.
Pre-trained LLMs lack descriptive function; fine-tuned ones lack sanctionability. Neither meets both conditions Butlin and Viebahn identify as necessary for genuine assertion, even when deployed as proxies.
OpenAI's Model Spec formalizes agent autonomy through structured scope records listing allowed tools, cost limits, and prohibited high-risk activities, mirroring software security's principle of least privilege while avoiding confirmation overload.
Show all 9 sources
Tool-enabled LLMs drift from user intent through silent tool chaining. Conversation analysis reveals insert-expansions—clarifying intent, scoping responses, enhancing appeal—as a formal framework for proactive user consultation that prevents misunderstanding instead of recovering from it.
Dung-style argumentation structures AI outputs as traversable attack/defense graphs, allowing users to identify and contest specific premises. Standard LLM outputs lack this structure, making it impossible to pinpoint which claims users actually reject.
A plausible design exists to extend LLM-agent modules with norm discovery, reasoning, and conformance by leveraging existing capabilities like world knowledge and Theory-of-Mind reasoning, though no implementation has been validated.
Research formalizes inter-agent thought sharing via sparse autoencoders that recover individual, shared, and private latent thoughts from hidden states. This approach detects alignment conflicts at the representational level before they manifest in language.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- AI Assertion
- Agentifying Agentic AI
- Securing Agentic AI: From Per-Action Checks to Trajectory Assurance
- A Comparative Study of MCP and A2A for Inter-Agent Coordination in LLM-Based Systems
- When Single-User-Oriented LLM-based Assistants Involve Others: A Scoping Review of Pathways, Risks, and Responses
- Toward an Organizational Science of Multi-Agent LLM Systems: Decoupling Who, How, and Which Algorithm
- Chatting with Bots: AI, Speech Acts, and the Edge of Assertion
- Harnessing the power of LLMs for normative reasoning in MASs