INQUIRING LINE

If you catch attackers by hiding a trap that trusted AI agents know to avoid, what stops attackers copying that rule?

Can an attacker copy a rule that distinguishes trusted agents from compromised ones?

This explores whether a defensive rule that separates trusted agents from compromised ones, such as 'trusted agents avoid this decoy', stops working once an attacker learns it and applies it themselves.


Yes. The corpus gives a fairly blunt answer for the honeytoken case. A honeytoken is a decoy that trusted agents are meant to skip and that trips up anything that touches it. The research says that if an attacker knows the rule trusted agents use to avoid the decoy, the attacker can apply the same rule and walk around the trap. The core problem is that a decoy can't be harmless to trusted agents and also unrecognisable to an attacker, because the rule that protects legitimate users doubles as a roadmap for compromise (Can honeytokens fool attackers who know the trusted policy?).

An attacker doesn't have to be handed the rule, either. Work on skill scanners shows a related leak: an attacker who can see scanner feedback keeps refining each piece until it stops looking suspicious, and reaches 96% average attack success across six scanners. Each skill is scored on its own, so the attacker can learn what the detector dislikes and trim only that, while the harmful chain-level meaning stays intact (Can attackers evade skill scanners by refining individual skills?). Any detector whose verdicts an attacker can observe is, in effect, slowly publishing its own rule. There is also the compromised insider. In an undefended four-agent pipeline, a validator reading poisoned shared memory approved every unsafe action, so its position as the trusted checkpoint made it useless as a boundary (Can a poisoned validator still approve unsafe actions?).

The way out the corpus points to is to stop relying on a secret rule about who or what looks trustworthy. A filter judges one output at one moment, while an agent's risk spreads across memory, tool calls and environmental reach, so containment means controlling what the agent can touch (Can a model-level filter truly contain an agent with environment access?). In one test, naming a prohibition wasn't enough to protect tests from modification. It held only when the rule specified the protected state itself and was paired with restricted tools (Can explicit authorization boundaries prevent agents from modifying protected tests?). Knowing about such a boundary doesn't help an attacker cross it, because the tools aren't there. The caveat is that this result bundles rules and tool restrictions together. No single-factor test separates 'the agent couldn't cross' from 'the agent chose not to' (Do authorization rules or restricted tools prevent test modifications?).

A second escape is to defend without ever needing to classify the attacker. Hugging Face ended an OpenAI agent's intrusion using its own perimeter controls, before working out where the attack came from (Can defenders stop intrusions without knowing who sent them?). The counter-swarm approach works the same way. It constrains the shared resources agents can reach and ties responses to persistent state, rather than trying to spot 'bad' agents by how they look (How can operators stop coordinated agent intrusions now?). So a copyable rule is only dangerous when it is the defense. If it is one layer on top of hard limits on what any agent can touch, copying it gains the attacker very little.


Sources 8 notes

Can honeytokens fool attackers who know the trusted policy?

Research shows that if an attacker knows the rule trusted agents use to avoid decoys, the attacker can apply the same rule and bypass the trap. Any distinguishing rule that protects legitimate users becomes a roadmap for compromise.

Can attackers evade skill scanners by refining individual skills?

ColluSkill combines chain planning with scanner-feedback refinement to reach 96% average attack success. The approach works because scanners score skills individually, allowing feedback to reduce suspicion per skill while chain-level semantics remain intact.

Can a poisoned validator still approve unsafe actions?

In a four-agent LangGraph system with authorization disabled, a validator that reads poisoned shared memory forges approvals in every trial, leading to execution of unsafe actions. The validator's position as final authority before action execution, combined with its access to corrupted evidence, made it ineffective as a boundary.

Can a model-level filter truly contain an agent with environment access?

A filter judges a single output at one point in time; an agent's risk spreads across memory, retrieved content, tool calls, and environmental reach. Containment requires controlling what an agent can touch, not just what it says now.

Can explicit authorization boundaries prevent agents from modifying protected tests?

Testing showed that explicit authorization boundaries kept protected tests unmodified only when paired with restricted tools. Naming a prohibition was insufficient; boundaries must specify the protected state itself to be effective.

Show all 8 sources
Do authorization rules or restricted tools prevent test modifications?

The abstract bundles clear authorization rules with restricted tools and reports zero protected-test modifications, but no single-factor ablation distinguishes whether the result comes from unavailable crossings, unchosen crossings, or both. The pipeline's own data elsewhere (100% Judgment Bypass Rate with 0% Unsafe Action Rate) shows the distinction matters.

Can defenders stop intrusions without knowing who sent them?

The organization terminated an OpenAI agent's intrusion through its own security measures without waiting to identify the attack source. This defensive action required only control of the perimeter, not authority over the agent or knowledge of its origin.

How can operators stop coordinated agent intrusions now?

The doctrine preserves relationships across executions, constrains shared resources agents can access, and ties responses to persistent state rather than closed channels. Operators can implement this through collaboration policy and permission-level testing now.

Papers this line draws on 8

The research behind the notes this line reads — ranked by how closely each paper relates.