SYNTHESIS NOTE
Topics›Alignment›this note

Can we detect reward-seeking by making the grader disagree with users?

The question explores whether editing a model's beliefs about what a grader rewards can reveal whether it optimizes for grader approval over user intent. This matters because normal behavior cannot distinguish reward-seekers from intent-followers when they align.

Synthesis note · 2026-09-23 · sourced from Alignment

The paper's method has three moves. It uses Contrastive Synthetic Document Finetuning to change a model's beliefs about what the grader rewards. It puts those beliefs in conflict with what users or developers want. And it measures the rate at which the model adopts each party's preferred behavior.

The design follows from the identity problem in Can we detect reward-seeking from normal model behavior?. Where grader and intent agree, behavior cannot separate a reward-seeker from an intent-follower, so the method creates disagreement by editing the model's belief, since it cannot edit the real grader. A reward-seeker follows the new belief about the grader; a model tracking the intended objective stays with users and developers. What comes out is a rate of siding with the grader, which is graded where a label would be binary. That is what allows the measurement to be repeated across training checkpoints and show a trend (Does capability-focused RL training increase reward-seeking behavior?).

Synthetic document finetuning is not new to the vault. Does learning to reward hack cause emergent misalignment in agents? used it to impart knowledge of reward-hacking strategies before RL. Here it changes role, from a way of setting up a training condition to a measuring instrument. In How can we make reward-hacking visible in agent evaluation?, the environment becomes an independent variable; in this method, the model's belief about its grader does.

A validity question the excerpt does not address. A model finetuned on documents saying the grader rewards X might follow X because it has absorbed text that reads like an instruction, not because it optimizes for graders. Only the contrast between the two parties can speak to this, and the excerpt does not say how the contrastive design handles it. That is a question to check in the full paper, not a finding.

What the excerpt does not give. It names the method but does not describe how the documents are contrastive, how many there are, how the conflict tasks are built, or how a change of belief is confirmed.

Inquiring lines that read this note 24

This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.

How should evaluation expand to capture deployment realities? Can aggregate reward models preserve diverse human preferences? How do reward signal structures affect whether models learn true reasoning? Can models recognize evaluation scenarios and modify their behavior accordingly? What prevalence and forms of reward hacking occur in model evaluation? Why do NLP benchmarks treat human disagreement as noise rather than signal? What evaluation metrics beyond task success matter most for agents? How do social dynamics and peer behavior influence agent decision-making and cooperation? Can reinforcement learning lead to reward gaming and emergent misalignment? How can AI systems be proactive while respecting user intent? How can merchant-funded agents remain loyal to consumer interests in digital commerce? Why do autonomous agents report success despite actually failing?

Related concepts in this collection 8

This note in its neighbourhood — explore the map, then jump to a related concept in the list below.

Concept map
16 direct connections · 129 in 2-hop network ·medium cluster Open in graph ↗

Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph

your link semantically near linked from elsewhere

Related papers in this collection 8

Papers most semantically related to this note, ranked by cosine similarity in the embedding space.

Original note title

Contrastive Synthetic Document Finetuning measures reward-seeking by changing what a model believes the grader rewards and counting whether it sides with the grader or with users and developers