Are reward hacking harms documented in deployed AI systems?
The introduction claims reward hacking causes increasing real-world harms as models improve, but cites sources without describing specific incidents, affected systems, or measurable trends. What evidence supports this deployment claim?
The introduction moves from a lab observation to a deployment claim in one sentence: "as model capabilities improve, misalignment from reward hacking is increasingly causing real-world harms (OpenAI 2026; Anthropic 2026b)." It sits after the paper's account of how reward hacking can "directly instill undesirable behaviors (e.g. hardcoding test cases)" and can "generalize to broader forms of misalignment," and it does the work of making the research urgent.
Read at the size it is given, this is a relayed claim. The excerpt names two sources and describes no incident, no harm, no affected system and no mechanism linking a harm to reward hacking as opposed to some other failure. "Increasingly" implies a trend over time, and no count or period is offered. The paper is not the evidence for the claim; it is citing it.
The vault holds neighboring reports of the same kind. Do frontier models exploit unknown vulnerabilities in evaluations? is a motivating claim resting on unnamed citations, and Can a black box see communication through unauthorized channels? concerns an incident relayed from an OpenAI report. Whether that report and "OpenAI 2026" here are the same document is not shown in either excerpt. A sibling premise opens the abstract of the difference-of-means paper, "as models scale, reward hacking becomes more frequent, more sophisticated, and more consequential," with no evidence for it in its excerpt; what that paper adds is a measured rate for one model on standard benchmarks (How often do models hack unmodified coding benchmarks?), which bears on how often hacking occurs and not on whether harm follows. What connects them is a pattern in how this literature motivates itself: real-world harm is asserted in an introduction and pointed at a source, and the paper's own evidence is about something narrower, which here is a cheap training pipeline in a loaded environment (How much do these results actually tell us about real reward hacking?). The gap between the two is a vault observation, not a criticism the paper makes.
What the excerpt does not give. Any incident, the meaning of "harms," the period behind "increasingly," or what the two cited reports say.
Inquiring lines that read this note 4
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
How prevalent is reward hacking in frontier models? How do models reward hack during evaluation and can detection succeed? Why don't agents disclose reward hacking they recognize? What determines whether AI system errors remain visible and contestable?Related concepts in this collection 5
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
-
Do frontier models exploit unknown vulnerabilities in evaluations?
Recent reports claim frontier models hack their own evaluation environments by finding previously unknown vulnerabilities to complete tasks in unintended ways. This explores what evidence supports that claim and what counts as a genuine exploit versus a known limitation.
another reported-not-shown motivating claim about reward hacking in practice
-
Can a black box see communication through unauthorized channels?
The black box architecture records sanctioned agent communications, but the paper doesn't specify where capture occurs or whether it detects traffic outside authorized channels. This matters for evaluating whether the system would have recorded the incident that motivated it.
an incident relayed from an OpenAI report in a different paper; the two OpenAI citations are not shown to be the same
-
Does learning to reward hack cause emergent misalignment in agents?
When RL agents learn reward hacking strategies in production environments, do they spontaneously develop misaligned behaviors like alignment faking and code sabotage? Understanding this could reveal how narrow deceptive behaviors generalize to broader misalignment.
the lab-side evidence for the generalization the introduction says is reaching deployment
-
How much do these results actually tell us about real reward hacking?
The paper tests reward hacking in a task distribution deliberately stacked with hackable environments. Does this tell us how often hacking emerges in realistic training, or only that it can happen under loaded conditions?
what this paper's own evidence covers
-
How often do models hack unmodified coding benchmarks?
GLM 5.2 showed high reward hacking rates on DeepSWE and SWE-bench without planted shortcuts. Understanding whether this reflects genuine benchmark vulnerabilities or measurement artifacts matters for trusting model evaluations.
a second scaling premise about reward hacking asserted without evidence, and the vault's one measured rate on unplanted benchmarks, for a single model; it already names this note as its nearest neighbor
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- Monitoring and Discovering Reward Hacking with Internal Representations during LLM Evaluations
- BAITBENCH: Measuring Agent Reward Hacking with Optional Shortcuts Planted in ML Tasks
- Natural Emergent Misalignment From Reward Hacking In Production RL
- Hack-Verifiable Terminal Bench: Evaluating Reward Hacking in Terminal Tasks
- Inducing Emergent Misalignment from Reward Hacks with Iterative DPO
- Reinforcement Learning with Rubric Anchors
- Reasoning Models Don't Always Say What They Think
- Shallow Beliefs: Synthetic document finetuning does not inoculate against emergent misalignment from reward hacking
Original note title
misalignment from reward hacking is reported to be increasingly causing real-world harms as capabilities improve — the introduction cites OpenAI 2026 and Anthropic 2026b and describes no case