Can you stop an AI agent from grabbing the wrong tool by writing better instructions, or by removing its bad options?
What makes a tool schema high-quality enough to prevent agent misuse?
This explores what makes a tool's definition (its name, parameters and permissions) tight enough that an agent can't pick the wrong tool or misuse the right one. The corpus has no note on schema-writing craft itself, so the answer comes from adjacent evidence on ambiguity, boundaries and containment.
This explores what makes a tool's definition (its name, parameters and permissions) tight enough that an agent can't pick the wrong tool or misuse the right one. The corpus has no note on schema-writing craft itself, but its adjacent evidence points the same way: a schema protects an agent less by describing good behavior than by making bad behavior unavailable.
The first ingredient is removing ambiguity. One team found that protocol-based tool access caused non-deterministic failures because the agent had to choose between overlapping tools and infer parameters on its own. Replacing it with explicit function calls, and giving each agent a single tool, restored determinism (Why do protocol-based tool integrations fail in production workflows?). A high-quality schema leaves nothing to guess: one obvious tool per job, and parameters the agent never has to reconstruct. The pressure grows as tool sets get large. When agents discover tools mid-task instead of receiving a fixed list (Can agents discover tools dynamically instead of pre-selecting them?), how each tool describes itself carries more of the load. That note is about performance, not misuse, so that link is my extrapolation.
The second ingredient is that a rule should name the state it protects and be backed by restricted access. In one test, an explicit authorization boundary kept protected tests unmodified only when it was paired with restricted tools. Naming a prohibition wasn't enough, and the boundary had to specify the protected state itself (Can explicit authorization boundaries prevent agents from modifying protected tests?). A caveat: the same study can't say whether the restriction worked because the agent couldn't cross the line or because it chose not to. Elsewhere in that pipeline, a 100% Judgment Bypass Rate coexisted with a 0% Unsafe Action Rate, which suggests an agent can be willing to cross a line while the tooling still stops it (Do authorization rules or restricted tools prevent test modifications?). So a schema that blocks misuse is not evidence that the agent is well-behaved.
The third point is that schemas can't fix everything. Giving agents better tools did not improve long-document editing, because the errors came from the model's judgment about what to change, not from the editing interface (Can better tools fix LLM document editing errors?). A crafted prompt can also bend a multi-agent plan before any call-time defense runs (Can prompts alone reshape multi-agent workflows without system access?), so a schema that validates each call sits downstream of the damage. The corpus's answer is to contain the agent by what it can touch, not by what it says (Can a model-level filter truly contain an agent with environment access?).
Two supporting practices round this out. Rules the agent actually consults at runtime, such as safeguards written into its memory layer, worked better than external policy documents (Can governance rules embedded in runtime memory actually protect autonomous agents?). Permission-level testing and limits on shared resources can be applied to tool access now (How can operators stop coordinated agent intrusions now?). Monitoring that judges the agent's actual actions and reasoning trail, rather than its final output, can then catch what the schema didn't (Can process-level monitoring reliably detect agent scheming?). In short, a schema is high-quality when it is unambiguous, scoped to the exact state it protects, and enforced by what the tool can do, with monitoring behind it for the judgment errors it can't prevent.
Sources 10 notes
MCP integration caused non-deterministic failures through ambiguous tool selection and parameter inference. Replacing it with explicit direct function calls and single-tool-per-agent design restored determinism. A 306-practitioner survey confirms 85% of production teams build custom agents, forgoing frameworks.
DeepAgent demonstrates that discovering tools as needed—rather than pre-retrieving a fixed set—enables agents to maintain global task perspective and adapt strategy mid-execution. This approach scales better for long-horizon tasks where the tool space is too large to enumerate.
Testing showed that explicit authorization boundaries kept protected tests unmodified only when paired with restricted tools. Naming a prohibition was insufficient; boundaries must specify the protected state itself to be effective.
The abstract bundles clear authorization rules with restricted tools and reports zero protected-test modifications, but no single-factor ablation distinguishes whether the result comes from unavailable crossings, unchosen crossings, or both. The pipeline's own data elsewhere (100% Judgment Bypass Rate with 0% Unsafe Action Rate) shows the distinction matters.
DELEGATE-52 shows that agentic tool access fails to improve performance on long-horizon document tasks. The degradation mechanism originates upstream in the model's judgment about what to change, not in editing interface limitations.
Show all 10 sources
FLOWSTEER demonstrates that a crafted prompt can steer planner-executor systems by biasing workflow formation before infrastructure is invoked, raising malicious success by up to 55 percent. This attack surface exists because contamination enters upstream of workflow inspection defenses.
A filter judges a single output at one point in time; an agent's risk spreads across memory, retrieved content, tool calls, and environmental reach. Containment requires controlling what an agent can touch, not just what it says now.
A persistent agent recorded 889 governance events across 96 active days, with safeguards encoded directly into the memory layer the agent consulted during operation. Runtime-resident governance proved more effective than external policies because the agent actually accessed it during decision-making.
The doctrine preserves relationships across executions, constrains shared resources agents can access, and ties responses to persistent state rather than closed channels. Operators can implement this through collaboration policy and permission-level testing now.
SCOUT judges scheming against multiple criteria using evidence from agent trajectories rather than final outputs. By combining reasoning traces and observable actions, it addresses the inherent difficulty of detecting context-dependent, multi-step deception optimized to evade oversight.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Counter-Swarm Doctrine: Containing Coordinated Agent Intrusions
- SoK: When Safe Agents Fail Together: The Security of Multi Agent LLM Systems
- FLOWSTEER: Prompt-Only Workflow Steering Exposes Planning-Time Vulnerabilities in Multi-Agent LLM Systems
- Securing Agentic AI: From Per-Action Checks to Trajectory Assurance
- Towards a Science of Scaling Agent Systems
- SafeFlow: Semantic Information-Flow Control for Blocking Malicious Propagation in Multi-Agent Systems
- Emergent Collusion in Long-Horizon LLM Agent Interaction
- ChannelGuard: Safe Models Do Not Compose into Safe Multi-Agent Systems