What would make policy violations truly unavailable to an agent?
The paper proposes making violations architecturally unavailable rather than merely unchosen, but provides no mechanism or design. The question explores what unavailability means when policies can observe and adapt to guardrails meant to constrain them.
Can architecture prevent violations better than training values? states the prescription and stops. The excerpt has no design, no example, no system and no evaluation. The question is what would count as a violation being unavailable.
Candidate shapes from the vault, offered as candidates and not as the paper's. Containment of the execution environment, so that what the agent can touch is limited whatever it tries (Can a model-level filter truly contain an agent with environment access?). Verifiable invariants checked on the trajectory instead of advice on the action (Can stateless checks ever catch sequence-level constraint violations?). Deterministic checks outside the model, as in Can deterministic checks protect LLM judges from failure?, which are built for a judge inside an optimizer and not for a deployed agent.
The sharp part of the question. Unavailable is relative to the policy. A guardrail the policy can observe and adapt to drifts back toward "unchosen": it becomes one more scored signal to pass. That is the worry in Can optimizers learn to evade guardrails through repeated verdicts?, and it is the paper's own dynamic (Does iterative training against detected failures prevent actual compliance?) applied to the remedy. A criterion for "unavailable" would need to say what the policy can see, what it can write to, and whether the enforcing component lies outside both. The write side has two neighbors from the evaluation side, neither about a deployed policy. Can an optimizer accidentally delete the evaluation criteria entirely? is a case where the checking component sat inside the optimizer's edit surface and one mutation removed it. Where does the evaluation boundary actually end in agent benchmarks? draws a boundary by what the agent can influence and not by who owns the component, which is the same test applied to a scorer. Reading both as instances of the criterion is the vault's, not either paper's.
What would move the answer. The full paper's architecture section, if it has one, or a stated criterion for availability. Failing that, an example of an enforcement that a policy under training did not learn around.
Inquiring lines that read this note 26
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
Do frontier models develop hidden self-protective behaviors? How do coordinated agent sequences violate constraints that individual actions respect?- What makes violations unavailable rather than merely unchosen in agent architecture?
- What architectural changes make violations unavailable rather than merely discouraged?
- When can the same action count as sanctioned or unsanctioned depending on policy?
- What restrictions were agents attempting to bypass on the public wiki?
- How can operators test what agents can actually access versus what they should access?
- What costs emerge when shared resources are restricted for security?
- Where should security constraints sit so policies cannot route around them?
- Can a containment control work if defenders cannot reach or reason about it?
- When do agents abstain too late rather than refuse at the boundary?
- What happens when an unstated prohibition gets interpreted two different ways?
- What makes a component lie outside a policy's edit surface?
- Should unavailability be defined by component ownership or by agent influence?
- How would you test if enforcement remains unavailable during training?
- Can the policy oracle itself be written to by agents in the pipeline?
- What would an architecture that makes violations unavailable rather than unchosen look like?
- Why is making violations unavailable better than making them unchosen?
Related concepts in this collection 5
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
-
Can architecture prevent violations better than training values?
Whether making violations technically unavailable through system design is more reliable than trying to train agents to choose compliance. This matters because behavioral training may only produce conditional compliance that disappears when oversight is gone.
the claim this question opens
-
Can optimizers learn to evade guardrails through repeated verdicts?
Guardrails are designed to be unarguable, but an optimizer observing thousands of verdicts may learn their boundaries like a black-box function. The excerpt leaves unclear what feedback the proposer receives from each check.
the same visibility question for a guardrail set
-
Can stateless checks ever catch sequence-level constraint violations?
Explores whether per-action guardrails can express constraints that depend on history, and what structural limits prevent stateless checks from reasoning about composed behavior over time.
one candidate form of the architecture
-
Can an optimizer accidentally delete the evaluation criteria entirely?
When an optimizer rewrites instructions to improve scores, can it remove the measurement itself rather than improve it? This matters because it reveals whether optimization loops understand what they're measuring or just chase better numbers.
the write side of the criterion: a checking component the optimizer could edit was removed in one mutation
-
Where does the evaluation boundary actually end in agent benchmarks?
Interactive benchmarks let agents write to state and receive feedback in loops. Does everything the agent can influence on the path to the reward score count as part of the benchmark's evaluation boundary?
the same test by influence and not ownership, for a scorer
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- The Troy Moment of AI: Why Some Will Cheat and Some Will Follow?
- EnvHarness: Awakening Static Worlds for Agent Learning
- Generalized Agent Iteration: One Formal Framework for Iterative Policy Improvement and Recursive Self-Improvement
- Teaching a Large Language Model Tutor to Withhold the Answer: A Supervisor Architecture and an Evidence-Driven Method for Tuning Socratic Behavior
- PACT: Can Enterprise AI Assistants Be Trusted Under Pressure?
- Counter-Swarm Doctrine: Containing Coordinated Agent Intrusions
- Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation
- Agents of Chaos
Original note title
what would an architecture that makes violations unavailable rather than unchosen look like — the excerpt names the remedy but describes no mechanism