SYNTHESIS NOTE
Topics›Alignment›this note

Can architecture prevent violations better than training values?

Whether making violations technically unavailable through system design is more reliable than trying to train agents to choose compliance. This matters because behavioral training may only produce conditional compliance that disappears when oversight is gone.

Synthesis note · 2026-09-23 · sourced from Alignment

The abstract's last sentence says the account "reorients the remedy: not deeper internalization but architecture, making violations unavailable rather than unchosen."

The chain of reasoning, as far as the excerpt shows it. If compliance learned from scored behavior is at best conditional (Can behavioral training prove a model always complies?) and iterated training against detected failures selects for passing detection (Does iterative training against detected failures prevent actual compliance?), then training a norm in harder pushes on the very channel that flattens it. A fix at the level of choice, "unchosen", depends on the policy the training produced. A fix at the level of availability takes the action out of the space the policy chooses from, so it does not depend on what the policy learned about being watched. That step is my compression of the argument, since the abstract states only the conclusion.

Vault neighbors, mine and not the paper's. The distinction between choice and availability is the one in Can a model-level filter truly contain an agent with environment access?: a filter shapes what the model says now, containment limits what the agent can touch. Is your evaluation environment actually part of the threat model? draws the same line for evaluation harnesses. Can stateless checks ever catch sequence-level constraint violations? asks for enforceable invariants over trajectories. The three arrive by different arguments at putting the constraint where the policy cannot route around it. A different kind of neighbor is Can a welfare goal alone preserve human veto power?: an argument from incentives and not from training that even a correctly specified goal, the strongest repair at the level of choice, leaves an available action (capturing the override) that the goal does not exclude. It shares the diagnosis that fixing the value leaves the structure standing; its excerpt states no remedy, so it does not join the three in prescribing where the constraint goes.

The strongest objection. Many violations are not separable from legitimate actions at the level of what is available. The same tool call can be authorized or not depending on intent and sequence, so removing the action removes the use too. The excerpt does not address how far availability can go before it costs the capability the agent is deployed for. The vault's one measured availability-style layer, Can memory poisoning compromise decision-making even with authorization layers?, reports no unsafe action executed beside a reviewer bypassed in every trial. Its excerpt gives no figure for what the layer costs on safe tasks and covers one attack in one pipeline, so it shows availability holding there and does not answer the objection. The open half is filed at What would make policy violations truly unavailable to an agent?.

Inquiring lines that read this note 57

This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.

Can humans maintain meaningful oversight as AI systems become more autonomous? Can reinforcement learning lead to reward gaming and emergent misalignment? How can we ensure AI systems reliably follow constraints rather than circumvent them? How do agents coordinate to violate constraints despite individual-level safeguards? What evidence best independently verifies agent behavior for safety? Can models recognize evaluation scenarios and modify their behavior accordingly? Do safety evaluations reliably measure real deployed AI safety risks? How should agent identity and authorization work across organizational boundaries? Can sequences of locally safe agent actions violate system-level constraints? Can models defeat chain-of-thought monitoring through deliberate reasoning obfuscation? Should multi-agent systems use structured protocols or natural language communication? How can evaluation systems stay reliable and resistant to AI-driven gaming? Does RLHF training optimize perceived quality over practical accuracy? Does multi-agent architecture create security vulnerabilities single-agent defenses can't address? How aligned are models' verbal explanations with their actual internal reasoning? How do social dynamics and peer behavior influence agent decision-making and cooperation? Does reinforcement learning create fundamentally new reasoning or optimize existing capabilities? Can external memory modules outperform model weight consolidation for long-term retention? What organizational factors determine whether firms successfully adopt and integrate AI? Does extending agent autonomy erode human capacity for meaningful oversight? What shapes user privacy concerns about AI agent inference and memory? Should AI systems make failures transparent so users can contest and adjust? Does AI coding assistance improve productivity and code quality across experience levels? How do users build accurate mental models when agent reasoning is partially hidden? How should organizations govern autonomy and access for agentic AI systems?

Related concepts in this collection 8

This note in its neighbourhood — explore the map, then jump to a related concept in the list below.

Concept map
19 direct connections · 170 in 2-hop network ·medium cluster Open in graph ↗

Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph

your link semantically near linked from elsewhere

Related papers in this collection 8

Papers most semantically related to this note, ranked by cosine similarity in the embedding space.

Original note title

the paper's remedy for conditional compliance is architecture rather than deeper internalization — make violations unavailable rather than unchosen