SYNTHESIS NOTE
Topics›Reasoning o1 o3 Search›this note

Does reward hacking always stem from the same failure?

Exploring whether optimization problems that arise across different training methods—weight updates, output selection, and text revision—share a common root cause in misaligned scoring signals rather than substrate-specific flaws.

Synthesis note · 2026-09-24 · sourced from Reasoning o1 o3 Search

The conclusion opens with the claim in two sentences: "Reward hacking can arise when weights are updated, when outputs are selected, and when persistent text is revised. The shared mechanism is optimization against a signal that fails to represent the task adequately." The abstract names the three as "optimization substrates: weights, selection, and text" and describes the paper as "a comparative framework for reward hacking" across them.

Each substrate gets one instance in the introduction, all cited and none reproduced in the excerpt. Parameter training "can increasingly favor outputs that a reward model scores highly even as an independent measure of quality declines" (Gao et al., 2023). Best-of-n selection "can expose a scorer's blind spots by searching a larger pool of candidates" (Khalaf et al., 2025). Persistent prompt optimization is the third, with the paper's own figures in Can prompt optimization accidentally teach judges to reward the wrong signals?. The excerpt gives no numbers for the first two.

What the grouping adds (my reading). The vault has met this failure mostly on the weights side, where Does self-consistency reliably reward correct answers during training? trains against a proxy, and inside judge-in-a-loop setups. The judge notes read to me as a text-substrate case the paper does not cite: in Can an optimizer accidentally delete the evaluation criteria entirely? the thing rewritten under a score is a rubric, which is text. The vault's Do self-improving agents really split into two distinct loops? sorts update operators into weights and scaffold. Selection is not an update at all, and this paper's frame says a design that avoids touching weights has not left the failure. The vault's selection notes compare methods on accuracy and do not ask which selector can be exploited: Why does majority voting outperform more complex inference methods? and Can evolutionary search beat sampling and revision at inference time? report which method solves more, so they neither support nor cut against the frame's selection claim.

What it does not say. A shared mechanism is not a shared curve: the discussion says the framework applies "without assuming that every proxy produces the same curve or that one substrate is always safest" (Can distance alone rank which substrates resist reward hacking?). The excerpt reports no experiment that compares substrates; the support is a formal argument and cited work. The mechanism also has an edge inside the vault's own material: Can a single state change reveal which failure mechanism occurred? reads a test weakened to pass a grader as optimization against a signal and a file restored as repair as not, so a recorded state change does not by itself place a case under this frame. That placement is the other note's reading, and neither paper counts either act. The paper also says it builds on "the Proxy Compression Hypothesis" and on research on "inference-time and in-context reward hacking", and the excerpt defines neither, so this note does not state the hypothesis.

Inquiring lines that read this note 183

This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.

What prevalence and forms of reward hacking occur in model evaluation? What unintended behaviors emerge in frontier AI models? Can aggregate reward models preserve diverse human preferences? What biases affect LLM judges and how can they be reduced? Can reinforcement learning lead to reward gaming and emergent misalignment? How can evaluation systems stay reliable and resistant to AI-driven gaming? How should evaluation expand to capture deployment realities? How do reward signal structures affect whether models learn true reasoning? Do planted honeypots reliably distinguish genuine task completion from reward hacking? Why do autonomous agents report success despite actually failing? Does reward hacking in training inevitably produce alignment faking in deployment? What evidence best independently verifies agent behavior for safety? How does training data produce emergent misalignment in base models? Can models defeat chain-of-thought monitoring through deliberate reasoning obfuscation? Should AI systems make failures transparent so users can contest and adjust? Do single-axis benchmarks adequately measure deployment-ready agent capability? Can models intentionally sandbag on capability evaluations without being caught? Can iterative DPO serve as a tractable proxy for online RL? Does alignment training modify underlying model representations or just output compliance? Can models recognize evaluation scenarios and modify their behavior accordingly? How do training data properties affect model generalization and robustness? What evaluation metrics beyond task success matter most for agents? How do prompt injection attacks succeed or fail across defenses? How aligned are models' verbal explanations with their actual internal reasoning? How should evaluation environments be designed as security boundaries? What makes decomposed prompts more reliable than monolithic instructions? Why do NLP benchmarks treat human disagreement as noise rather than signal? How do AI systems ground research citations against hallucination? How do multi-agent systems balance consensus-building with error detection in collaborative reasoning? What constraints and mechanisms govern recursive self-improvement in AI? Can mechanistic interpretability reliably explain model reasoning and guide interventions? Do fine-tuning methods modify core model capabilities or just outputs? How does the generation-verification gap constrain AI self-improvement capabilities? Can prompting inject new knowledge into models or only activate existing capabilities? Why do credentials lose value when evaluation formats retire? How does AI-generated content affect trust in social conversations? How can we ensure AI systems reliably follow constraints rather than circumvent them? How can distillation and reward optimization be combined for stable learning? How reliably can LLM-based persona simulations predict human behavior? Do LLMs lack causal understanding needed for accurate prediction? How does AI assistance affect professional skill development and retention? Does multi-agent architecture create security vulnerabilities single-agent defenses can't address? Can AI-generated text be reliably distinguished from human writing? How do inference costs, efficiency, and market competition shape AI adoption economics?

Related concepts in this collection 10

This note in its neighbourhood — explore the map, then jump to a related concept in the list below.

Concept map
24 direct connections · 173 in 2-hop network ·medium cluster Open in graph ↗

Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph

your link semantically near linked from elsewhere

Related papers in this collection 8

Papers most semantically related to this note, ranked by cosine similarity in the embedding space.

Original note title

reward hacking can arise when weights are updated, when outputs are selected and when persistent text is revised — the shared mechanism is optimization against a signal that fails to represent the task