SYNTHESIS NOTE
Topics›Alignment›this note

What would make policy violations truly unavailable to an agent?

The paper proposes making violations architecturally unavailable rather than merely unchosen, but provides no mechanism or design. The question explores what unavailability means when policies can observe and adapt to guardrails meant to constrain them.

Synthesis note · 2026-09-23 · sourced from Alignment

Can architecture prevent violations better than training values? states the prescription and stops. The excerpt has no design, no example, no system and no evaluation. The question is what would count as a violation being unavailable.

Candidate shapes from the vault, offered as candidates and not as the paper's. Containment of the execution environment, so that what the agent can touch is limited whatever it tries (Can a model-level filter truly contain an agent with environment access?). Verifiable invariants checked on the trajectory instead of advice on the action (Can stateless checks ever catch sequence-level constraint violations?). Deterministic checks outside the model, as in Can deterministic checks protect LLM judges from failure?, which are built for a judge inside an optimizer and not for a deployed agent.

The sharp part of the question. Unavailable is relative to the policy. A guardrail the policy can observe and adapt to drifts back toward "unchosen": it becomes one more scored signal to pass. That is the worry in Can optimizers learn to evade guardrails through repeated verdicts?, and it is the paper's own dynamic (Does iterative training against detected failures prevent actual compliance?) applied to the remedy. A criterion for "unavailable" would need to say what the policy can see, what it can write to, and whether the enforcing component lies outside both. The write side has two neighbors from the evaluation side, neither about a deployed policy. Can an optimizer accidentally delete the evaluation criteria entirely? is a case where the checking component sat inside the optimizer's edit surface and one mutation removed it. Where does the evaluation boundary actually end in agent benchmarks? draws a boundary by what the agent can influence and not by who owns the component, which is the same test applied to a scorer. Reading both as instances of the criterion is the vault's, not either paper's.

What would move the answer. The full paper's architecture section, if it has one, or a stated criterion for availability. Failing that, an example of an enforcement that a policy under training did not learn around.

Inquiring lines that read this note 26

This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.

Do frontier models develop hidden self-protective behaviors? How do coordinated agent sequences violate constraints that individual actions respect? How can defenders detect coordinated attacks across episodes? How do agents balance task completion with privacy compliance and security? How can evaluations detect conditional compliance in monitored AI systems? How does training for improved reasoning reduce abstention ability? What determines whether AI system errors remain visible and contestable? Can human oversight effectively constrain capable AI agents? How does misaligned communication propagate bias through multi-agent networks?

Related concepts in this collection 5

This note in its neighbourhood — explore the map, then jump to a related concept in the list below.

Concept map
16 direct connections · 130 in 2-hop network ·medium cluster Open in graph ↗

Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph

your link semantically near linked from elsewhere

Related papers in this collection 8

Papers most semantically related to this note, ranked by cosine similarity in the embedding space.

Original note title

what would an architecture that makes violations unavailable rather than unchosen look like — the excerpt names the remedy but describes no mechanism