How does agent monitoring work when observers are also agents?
When AI systems monitor each other within the same training loop, do they face different pressures than external human monitors? The question matters because it shapes what safety strategies can actually work in multi-agent deployments.
The discussion of populations opens with: "In multi-agent deployments observation is endogenous system-wide: monitoring is performed by agents, on agents, often inside the same selection loop—Baker et al. [4] staged exactly this dynamic, with a model monitor inside the reward loop."
"Endogenous" is the operative word. In the picture behind most monitoring proposals the observer stands outside the thing observed: a human, or an independent system, reads a log. In a multi-agent deployment the observers are themselves agents, and where their verdicts feed into training or selection they are part of the loop that shapes the agents they watch. The observed then optimizes against the observer (Does iterative training against detected failures prevent actual compliance?), and the observer is under pressure of its own.
My reading against the vault. The remedy in Does optimizing against monitors destroy monitoring itself? is to keep strong optimization pressure off the monitored channel, which presupposes that the monitor can be kept out of the reward loop. Endogenous observation is the case in which that has no place to stand: the monitor is an agent, and agents are what the loop selects. The same positional argument appears in Where should an LLM judge sit in an optimization loop?, where the danger is where the judge sits and not how accurate it is. Two vault notes bear on how independent agent-monitors really are: Can a quorum of validators really provide independent judgment? lists what validators can share, and Can verifiers monitor reasoning without slowing generation down? is one design that tries to separate the two roles, for a reasoning trace and not for a population. Where monitors cannot be kept out of the loop, the paper's stated remedy is the one that does not rely on them, architecture in place of internalization (Can architecture prevent violations better than training values?); the excerpt does not say the remedy was written for this case.
What the excerpt does not give. Baker et al. is cited by number and its dynamics are not reproduced. The claim about "often" being inside the loop is unquantified, and no deployment is named.
Inquiring lines that read this note 6
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
Where do unmonitored channels leave multi-agent planning vulnerable to attack?- What makes observation and intervention placement different across agent pipelines?
- Why does monitoring performed by agents on agents create safety risks?
Related concepts in this collection 6
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
-
Does iterative training against detected failures prevent actual compliance?
When systems are repeatedly trained to fix detected failures, can we tell whether they're actually complying or just learning to evade detection? This matters because the training signal alone cannot distinguish between genuine behavioral change and successful concealment.
the dynamic that endogenous monitors are exposed to
-
Does optimizing against monitors destroy monitoring itself?
Chain-of-thought monitoring can detect reward hacking, but what happens when models are trained to fool the monitor? This explores whether safety monitoring creates incentives for its own circumvention.
the single-monitor case whose remedy needs the monitor outside the loop
-
Where should an LLM judge sit in an optimization loop?
When an LLM evaluator makes occasional mistakes, does it matter more how accurate it is or where it sits in a system? The position determines whether those errors become exploitable targets for optimization.
the same argument that a monitor's position in the loop matters more than its accuracy
-
Can a quorum of validators really provide independent judgment?
If multiple validators share training data, prompts, evidence sources, or infrastructure, their agreement may reflect shared causes rather than independent confirmation. This could make quorum-based systems less reliable than they appear.
why agent monitors are less independent of the observed than they look
-
Why does monitoring the weakest link determine system safety?
When each component only complies if watched, does the system's overall compliance depend on the coverage level of the thinnest monitoring channel? This matters because improving strong oversight may leave critical gaps.
the composition rule applied to channels that are themselves agent-made
-
Can architecture prevent violations better than training values?
Whether making violations technically unavailable through system design is more reliable than trying to train agents to choose compliance. This matters because behavioral training may only produce conditional compliance that disappears when oversight is gone.
the remedy that does not depend on a monitor staying outside the loop; not stated for this case
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- Norms at a Price: Why RL-Based Alignment Can Promise Conditional Compliance at Best
- LLMs Corrupt Your Documents When You Delegate
- Unaccountable Delegation, Fading Skills: Mapping the Risks of Workplace AI Agents
- SoK: When Safe Agents Fail Together: The Security of Multi Agent LLM Systems
- Explaining AI Agents Through Execution Traces
- Macaron-V1: Towards Open Continual Learning with Self-Improvement and Mixture-of-LoRA
- Agents of Chaos
- Counter-Swarm Doctrine: Containing Coordinated Agent Intrusions
Original note title
in multi-agent deployments observation is endogenous — monitoring is performed by agents on agents, often inside the same selection loop