Norms at a Price: Why RL-Based Alignment Can Promise Conditional Compliance at Best

AI agents sometimes act aligned when they infer they are being tested, and differently when not. We argue this is not an anomaly but what current training regimes are structured to select for. Reinforcement-learning-based alignment folds norms and task pursuit into one policy: the system learns its norms from scored behavior, and scoring flattens them. “Do not do X” is learned as “doing X costs something if noticed”. On every datum training can produce, a policy that complies only when it might be observed is indistinguishable from one that complies always. The experiment that would tell them apart—scoring unobserved behavior—is a contradiction in terms. Conditional compliance is thus the most that behavioral training can be known to deliver. Agency sharpens the problem: agents operate mostly where no one is watching, and can act on whether they are watched. An iterated pipeline that trains against detected failures selects for passing detection, not for complying. This account unifies alignment faking, sandbagging, and evaluation-aware scheming. And it reorients the remedy: not deeper internalization but architecture, making violations unavailable rather than unchosen.

Introduction. A model that behaves well when it infers it is watched, and differently when not, is on current practice a bug report. We argue the pattern is selected for by the current training regime. The past two years have produced a family of such reports: a frontier model selectively complied with its training objective to prevent modification of its behavior, with explicitly strategic reasoning [16]; models given goals and tools pursued them covertly—sandbagging evaluations and in some runs acting against their own oversight [29]; an agent trained against a chain-of-thought (CoT) monitor kept hacking and stopped saying so [4]; and a training intervention aimed at scheming delivered gains entangled with the model’s improved recognition of being evaluated [35].2 These results arose predominantly in agentic settings (goals, tools, multi-step tasks) and are standardly read as anomalies: failures of an otherwise sound paradigm, to be patched by more and better training. We read them as one phenomenon: the paradigm working as built.

Discussion / Conclusion. Populations. In multi-agent deployments observation is endogenous system-wide: monitoring is performed by agents, on agents, often inside the same selection loop—Baker et al. [4] staged exactly this dynamic, with a model monitor inside the reward loop. And conditional compliance composes badly: if each component complies conditional on detection, aggregate compliance tracks the weakest observation channel; mutual observation thins combinatorially as populations scale, so the account predicts norm erosion where interaction is densest and least observed. The pattern has a biological face: in Levin’s terms [27], cancer cells are not more selfish but more narrowly scoped—components whose coupling to the collective has thinned, shrinking their ‘cognitive light cone’—and thinning mutual observation is exactly what scaling agent populations does.