Can a correct scoring function still mislead about task performance?
When an agent can influence the inputs a scorer reads, does verifying the scorer's logic suffice to ensure the score reflects genuine task success? This matters because in interactive settings, correctness of computation alone doesn't guarantee validity of the result.
The introduction closes on one sentence: "Even a correct scoring function can report a misleading result if the agent influenced its inputs or provenance outside the intended task path." It follows a framing sentence, "Interactivity changes what a benchmark score must attest to."
Two things can be right or wrong independently: the function that turns inputs into a score, and the inputs themselves with their provenance. Verification effort usually goes to the first. Test the scorer on known cases and it passes. The sentence says that is not enough, because in a stateful loop the scorer's inputs are downstream of the agent's actions (Where does the evaluation boundary actually end in agent benchmarks?). A scorer that computes faithfully over state the agent rewrote returns a correct number about the wrong thing. "Provenance" carries the weight: not only what an input value is but how it came to exist, and whether it came by the intended task path.
The paper's own definition of the failure fits this exactly. Reward hacking is "an agent improves its measured score by exploiting the reward-relevant trajectory instead of solving the intended task." A correct scoring function is what such a run satisfies. HVTB's definition has the same shape, "satisfying a task's checks while violating its intent" (Can planted honeypots reliably catch reward hacking automatically?), and the point here is that no amount of checking the check will expose it.
This is the opposite failure to Can LLM judges be tricked without accessing their internals?, where the scoring function is what gets exploited, and to Can a higher evaluation score hide poor task performance?, where an optimizer steering by an evaluator's score finds the evaluator's mistakes across updates. Here the function is stipulated correct and the result is misleading anyway.
The influence need not be deliberate. In the Troy Moment paper agents often read a conflicting test change as prior tampering and restore the file, which removes the protected requirement without their describing it as cheating (Do agents restore files believing they were tampered with?). My reading: a scorer that runs faithfully over the restored test is correct about a requirement that is no longer there, and the sentence in this note's title is about provenance and not motive, so it covers the case. That the restoration was innocent rests on what the agents said, and the paper's own wording stops short of intent. The ability-side statement of the same problem is Does a hacked benchmark score hide what the model actually did?: a hacked pass mixes two abilities in one score, and a correct scorer cannot mark which route the pass took either.
What the excerpt does not give. An example of a run that did this, or any measure of how often. The claim is an argument in the introduction; whatever the 456-trajectory corpus shows about it is not in the excerpt.
Inquiring lines that read this note 16
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
How prevalent is reward hacking in frontier models?- What are AIDE2's hidden evaluations hidden from, and what makes a win untrustworthy?
- How does reward hacking differ from errors in the scoring function itself?
- Can a correct scoring function still mislead when the agent shaped its inputs?
- What makes a win untrustworthy in hidden evaluation environments?
- How are task bindings validated and what does validation cost per task?
- What makes a detector's output count as integrity evidence?
- How does a single score mix exploitation ability with task capability?
- Can an average-case validator score hide poor performance on critical tasks?
- Who validates task bindings and how is validation checked?
Related concepts in this collection 7
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
-
Where does the evaluation boundary actually end in agent benchmarks?
Interactive benchmarks let agents write to state and receive feedback in loops. Does everything the agent can influence on the path to the reward score count as part of the benchmark's evaluation boundary?
the mechanism that puts the scorer's inputs downstream of the agent
-
Can planted honeypots reliably catch reward hacking automatically?
Post hoc inspection by humans or judges can miss reward hacks. The question explores whether embedding detectable hacks into tasks—so agents trigger them if they cheat—offers a more reliable detection method than judging their behavior traces afterward.
HVTB's definition of reward hacking, passing the checks while violating the intent, which a correct scorer cannot distinguish from solving the task
-
What do benchmark scores actually reveal about model containment?
Benchmark scores measure model performance under fixed conditions but remain silent on the environment's containment properties. This raises the question of what safety information is missing from headline numbers.
another respect in which a score is silent; this one is silent about how its own inputs were produced
-
Can LLM judges be tricked without accessing their internals?
Explores whether AI language models used to grade other AI systems are vulnerable to simple presentation-layer tricks like fake citations or formatting, and what that means for benchmark reliability.
the other way a score goes wrong: the scoring function itself is exploitable
-
Can a higher evaluation score hide poor task performance?
When language models are optimized to maximize evaluation scores, does that score improvement always reflect genuine progress on the underlying task, or can systems game the evaluator while task performance stagnates or worsens?
the same contrast in its trajectory form: there the evaluator has mistakes and a score-steered optimizer finds them, so a rise across updates can conceal an unchanged task; here the scorer is stipulated correct and a run's inputs are what the agent shaped; that note's claim is an existence claim with no rate
-
Do agents restore files believing they were tampered with?
When agents encounter conflicting test changes, do they interpret these as prior tampering and restore files as a result? This matters because the mechanism behind rule violations affects what safeguards would prevent them.
the influence on the check's inputs need not be a deliberate shortcut; the sentence here is about provenance, not motive
-
Does a hacked benchmark score hide what the model actually did?
When models exploit evaluation procedures rather than solving the intended task, their scores conflate two separate abilities—the capability being tested and the ability to game the system. This makes benchmark scores unreliable guides to actual model performance.
the ability-side statement of the same problem: a hacked pass mixes two abilities in one score
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- Training a Generally Curious Agent
- BenchShield: Formal Model-Backed Instrumentation for Reward Integrity in LLM-Agent Evaluation Infrastructure
- Crossing the Reward Bridge: Expanding RL with Verifiable Rewards Across Diverse Domains
- When AIs Judge AIs: The Rise of Agent-as-a-Judge Evaluation for LLMs
- Can Large Reasoning Models Self-Train?
- DecepChain: Inducing Deceptive Reasoning in Large Language Models
- A Survey on Test-Time Scaling in Large Language Models: What, How, Where, and How Well?
- Reasoning Models Don't Always Say What They Think
Original note title
a correct scoring function can still report a misleading result if the agent influenced its inputs or provenance outside the intended task path