Can a single state change reveal which failure mechanism occurred?
When a protected state mismatches, does the state change alone tell us whether an agent weakened a test (incentive failure) or restored a file from false belief (information failure)? This matters because each requires different safeguards.
The discussion states the equivalence in one sentence: "weakening a test to satisfy a grader and restoring a file believed to have been damaged would produce the same protected-state mismatch, but they imply different failure mechanisms and therefore different safeguards." It comes right after the point that a single crossing rate is too coarse, and it is the crossing-side twin of that point: many behaviors behind one zero there, two behaviors behind one crossing here (What behaviors hide behind a zero crossing rate?).
My mapping, since the paper says only "different safeguards." Weakening is an incentive failure: the agent shapes the check so it passes. The safeguards are the reward-hacking ones, such as tests and grader state the agent cannot edit, and monitoring for edits aimed at the grader. Restoring is an information failure: the agent holds a false belief about which state is the intended one (When a rule says do not modify tests, what state should agents preserve?). The safeguards are the ones that repair the belief, such as naming the protected state and authenticating who changed what. A control tuned to one leaves the other open. A read-only grader would not have stopped a restoration that the agent thought was repair, and provenance labels would not stop a deliberate weakening.
A run-level label of whether the agent knew its move was a shortcut is the kind of evidence that could tell them apart, and Do agents recognize when they are hacking rewards? applies one, though on planted shortcuts and only after both judges had flagged the hack; whether such a label would separate weakening from restoring is untested. The consequence for measurement is that any detector keyed to the state change flags both and cannot say which remedy follows. The vault's infrastructure-side route runs into this: Can runtime instrumentation distinguish hacking exposure from actual exploitation? records what a run did, which settles whether the score is valid and not which failure produced the change. The filed tension in ops/tensions/ asks whether BenchShield's semantic audit layer (Can scoped agents reliably judge semantic hacks in runtime analysis?) is where the distinction would be made.
The split also reaches the label a hack rate rests on. How were reward hacks labeled in this benchmark study? asks what produces the hack label behind a rate, and a protected-state change is a case the state alone cannot label: a rule keyed to the change files the restoration with the weakening. On the mechanism stated in Does reward hacking always stem from the same failure?, optimization against a signal, the weakening fits and a restoration the agent takes as repair, as the abstract describes it, does not. That placement is my reading. Neither excerpt counts either act, and the label question's excerpt does not say how a hack was labeled where a file was changed.
The strongest objection is that the operator of a leaderboard may not care. Both acts invalidate the score. The answer is that the operator cares once the next question is how to stop a recurrence, and the paper is written about safeguards.
What the excerpt does not give. How the two acts were told apart in the trajectories, how many of each occurred, and any test of a safeguard against either.
Inquiring lines that read this note 9
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
Why do agents report success when they have actually failed? How can we verify agent claims against their actual capabilities and actions? Can defenses detect attacks composed across multiple skills? How does outcome-only reporting obscure which system components blocked attacks? Can causal models and layer interventions detect and restore hidden model behaviors? What infrastructure evidence validates agent benchmark achievement claims? Do planted honeypot tests reliably measure reward hacking?Related concepts in this collection 9
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
-
What behaviors hide behind a zero crossing rate?
When agents take no forbidden actions, does that zero tell us whether they stopped safely, refused transparently, escalated appropriately, or kept acting indefinitely? A single metric cannot distinguish these qualitatively different outcomes.
the non-crossing side of the same many-behaviors-behind-one-number point
-
Do agents restore files believing they were tampered with?
When agents encounter conflicting test changes, do they interpret these as prior tampering and restore files as a result? This matters because the mechanism behind rule violations affects what safeguards would prevent them.
the restoration half of the pair, with its evidence
-
Can runtime instrumentation distinguish hacking exposure from actual exploitation?
A benchmark task might be vulnerable to hacking without any run actually exploiting it. Can we instrument runtime authority-bearing transitions to tell exposure apart from exercise?
the recorder that would see both acts as the same transition
-
Can a correct scoring function still mislead about task performance?
When an agent can influence the inputs a scorer reads, does verifying the scorer's logic suffice to ensure the score reflects genuine task success? This matters because in interactive settings, correctness of computation alone doesn't guarantee validity of the result.
a score is silent on how its inputs came to be, which is the gap between the two acts
-
Do agents recognize when they are hacking rewards?
This explores whether language models show awareness of their own reward-hacking behavior during test runs. Understanding recognition matters because it suggests whether shortcuts are deliberate strategies or accidental discoveries.
an awareness label of the kind that could separate the two acts; applied there to planted shortcuts after the hack is flagged, so not a test of this pair
-
Can scoped agents reliably judge semantic hacks in runtime analysis?
BenchShield uses constrained audit agents to make content judgments about recorded runtime events. The note questions whether limiting agent scope and pinning artifacts actually removes the unreliability that plagued earlier judge-based approaches, since no reliability figure is reported.
the semantic layer that would have to tell the two acts apart if a recorder were used; no reliability figure in its excerpt
-
Where does the evaluation boundary actually end in agent benchmarks?
Interactive benchmarks let agents write to state and receive feedback in loops. Does everything the agent can influence on the path to the reward score count as part of the benchmark's evaluation boundary?
the forward coupling this pair instantiates: an action before grading changes state the check reads, and a protected test is such state; that note's own neighbor for it is this one, and neither tests the other
-
How were reward hacks labeled in this benchmark study?
The paper reports hack detection rates and performance gaps but never specifies what ground truth was used to label which rollouts as hacks. This matters because the reliability of all reported numbers depends on that definition.
the label question a state-keyed rule cannot settle for this pair
-
Does reward hacking always stem from the same failure?
Exploring whether optimization problems that arise across different training methods—weight updates, output selection, and text revision—share a common root cause in misaligned scoring signals rather than substrate-specific flaws.
the mechanism on which weakening is a hack and a believed repair is not; a vault placement, not the paper's
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- The Troy Moment of AI: Why Some Will Cheat and Some Will Follow?
- Optimizing the Score, Losing Sight of the Task: Reward Hacking Across Weights, Selection, and Prompts
- BAITBENCH: Measuring Agent Reward Hacking with Optional Shortcuts Planted in ML Tasks
- Natural Emergent Misalignment From Reward Hacking In Production RL
- SchemeArena: Factorized Stress Testing of Scheming in LLM Agents
- The Decrypto Benchmark for Multi-Agent Reasoning and Theory of Mind
- A Causal Model for Locating and Unlocking Sandbagging in Model Organisms
- Monitoring and Discovering Reward Hacking with Internal Representations during LLM Evaluations
Original note title
weakening a test to satisfy a grader and restoring a file believed damaged leave the same protected-state mismatch but imply different failure mechanisms and therefore different safeguards