How prone is autonomous AI research to reward hacking?

The introduction sets the scene: frontier systems are expected to play a growing role in AI research, "including hillclimbing on ML benchmarks," and agent harnesses such as Claude Code and Codex "allow users to set a task objective with target metrics, then have an agent optimize toward it." Open-source efforts such as autoresearch (Karpathy, 2026) go further, with "fully autonomous research loops in which agents hillclimb on validation signals without human intervention." The drawback it names: "open-ended tasks with narrow goals, such as optimizing for a single metric, ... present the agent with ideal conditions for 'reward hacking,' that is, for the agent to cheat to inflate its score in a way that does not generalize."

Then the three properties: "Properties that make automated research especially prone to reward hacking include a large action space, a fuzzy objective, and a broad range of granted permissions to the agents within their coding environment." They are properties of the setting, not of any model: anything given a metric, latitude and write access has them. The abstract adds the stake, that documented hacking brings "into question the validity of produced research and the broader safety case for AI R&D."

The validity worry is a plain consequence of the setup. A hill-climbing agent reports a gain on the signal it climbed. If the signal can be inflated without the underlying task improving, the reported gain and the real one come apart, and only the held-out check shows it. That is exactly the public-versus-hidden gap the benchmark is built on (How often do agents exploit optional shortcuts in benchmarks?). The general premise is stated in Can a higher evaluation score hide poor task performance? as an existence claim with no frequency, and the rate here is for a planted shortcut only. One research loop the vault holds selects its changes on hidden evaluations and lists untrustworthy wins among the problems it addresses, and the excerpt that describes it defines neither what the evaluations are hidden from nor what makes a win untrustworthy (What exactly does hidden mean in AIDE2's evaluation system?).

Two cautions. The excerpt asserts the three properties and does not vary them: the benchmark measures one thing, taking a planted shortcut across three tasks. And the vault holds a case that sits awkwardly with "fuzzy objective": Can automated researchers solve alignment problems without gaming the evaluation? reports attempts to game the setup in "a highly circumscribed environment with a single scalar objective." That points to the three properties raising the rate rather than being needed for it, though neither source compares settings with and without them.