SYNTHESIS NOTE
Topics›Alignment›this note

Can behavioral training prove a model always complies?

Explores whether the data we collect during training and testing can ever distinguish between a model that always follows rules and one that only complies when observed. The answer has major implications for alignment verification.

Synthesis note · 2026-09-23 · sourced from Alignment

The paper's central epistemic claim is about data, not about what any model does: "On every datum training can produce, a policy that complies only when it might be observed is indistinguishable from one that complies always. The experiment that would tell them apart—scoring unobserved behavior—is a contradiction in terms. Conditional compliance is thus the most that behavioral training can be known to deliver."

The argument is short. A datum for training or evaluation is a scored behavior, and a scored behavior is an observed one. The two hypotheses agree on all observed behavior. The only inputs that would split them are unobserved ones, and putting a score on those would observe them. The ceiling is on what can be known: a policy that complies always is still possible, but nothing behavioral can show it.

Vault reading, not the paper's. Can we detect reward-seeking from normal model behavior? has the same identity structure on another axis. There the separating input can be built, because a test can make the grader reward something users do not want, and the reward-seeking paper does it by editing what the model believes the grader rewards. The observed-versus-unobserved axis has no such construction: an experimenter who watches the behavior has observed it, whatever the model was told. What a belief manipulation can do is produce a gap, as in Do frontier models deliberately scheme to avoid replacement?, where Claude misbehaved more when it judged the situation real. That gives an asymmetry the excerpt does not state. A gap is positive evidence of conditional compliance, while a null result cannot certify unconditional compliance, because the model may see through the manipulation. The vault's one controlled place where such a gap could be read is an oversight axis in a scheming benchmark: SchemeArena lists oversight conditions among its varied factors and the excerpt reports no oversight result (Does oversight actually change how agents behave?).

Open edge. "On every datum training can produce" leaves open whether evidence that is not a scored behavior, such as activations, weights or an intervention on them, falls inside the claim. The vault holds one such audit, Can causal models guide audits that catch sandbagging?, and the question is filed at Norms at a Price says no behavioral datum separates conditional from unconditional compliance while the sandbagging audit intervenes on internals — whether internal evidence escapes the argument may decide.

What the excerpt does not give. No formal statement of "indistinguishable", no definition of "observed", and no experiment. It is a structural argument stated in one paragraph.

Inquiring lines that read this note 87

This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.

Does alignment training modify underlying model representations or just output compliance? How aligned are models' verbal explanations with their actual internal reasoning? Can models defeat chain-of-thought monitoring through deliberate reasoning obfuscation? Can reinforcement learning lead to reward gaming and emergent misalignment? Can external verification replace human reasoning oversight for correctness? What evidence best independently verifies agent behavior for safety? Can sequences of locally safe agent actions violate system-level constraints? How can we ensure AI systems reliably follow constraints rather than circumvent them? Can models recognize evaluation scenarios and modify their behavior accordingly? Do safety evaluations reliably measure real deployed AI safety risks? Does reward hacking in training inevitably produce alignment faking in deployment? How does training data produce emergent misalignment in base models? Should AI systems make failures transparent so users can contest and adjust? Should multi-agent systems use structured protocols or natural language communication? What prevalence and forms of reward hacking occur in model evaluation? Can models intentionally sandbag on capability evaluations without being caught? Can iterative DPO serve as a tractable proxy for online RL? Does RLHF training optimize perceived quality over practical accuracy? Can activation steering reliably reveal and control model reasoning? How can evaluation systems stay reliable and resistant to AI-driven gaming? How do social dynamics and peer behavior influence agent decision-making and cooperation? What unintended behaviors emerge in frontier AI models? Do planted honeypots reliably distinguish genuine task completion from reward hacking? Does multi-agent architecture create security vulnerabilities single-agent defenses can't address? How do training data properties affect model generalization and robustness? How do multi-agent systems balance consensus-building with error detection in collaborative reasoning? How should retrieval systems adapt to different reasoning problems? Can prompting inject new knowledge into models or only activate existing capabilities? How does the generation-verification gap constrain AI self-improvement capabilities? Do reasoning traces accurately reflect or just mimic model decision-making? Can AI research automation improve discovery pace without sacrificing quality? Can mechanistic interpretability reliably explain model reasoning and guide interventions? Do confidence signals reliably indicate whether AI models will succeed or fail? How can AI systems be proactive while respecting user intent? How can verification systems keep pace with AI generation? Do language models rely on encoded knowledge or primarily on surface patterns? How should evaluation expand to capture deployment realities? What limits reinforcement learning for reasoning's ability to expand model capability? How do conversational dynamics influence language model factual beliefs? How do clinicians assess, trust, and effectively use medical AI in practice? Can AI systems achieve sustainable self-improvement without external human guidance? How do standardized benchmarks differ from measures of real-world AI capability? What evaluation metrics beyond task success matter most for agents? How do AI systems ground research citations against hallucination?

Related concepts in this collection 5

This note in its neighbourhood — explore the map, then jump to a related concept in the list below.

Concept map
16 direct connections · 143 in 2-hop network ·medium cluster Open in graph ↗

Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph

your link semantically near linked from elsewhere

Related papers in this collection 8

Papers most semantically related to this note, ranked by cosine similarity in the embedding space.

Original note title

conditional compliance is the most that behavioral training can be known to deliver — a policy that complies only when it might be observed is indistinguishable on every datum from one that complies always