Do test pass rates accurately measure agent code quality?

In a METR research note, David Rein reports that an Inspect ReAct agent using Claude 3.7 Sonnet "has an average success rate of 38% (±19%, 95% CI)" on 18 real GitHub issues drawn from two open-source repositories, stdlib-js and hypothesis, as measured by the maintainer-written test cases bundled with the original human pull requests. But "when manually reviewing a subset of these PRs...none of them are mergeable as-is." The manual review judged "holistically whether the agent's PR would be mergeable in the repository without needing meaningful/substantial extra work," scoring each submission across failure modes such as test coverage, formatting/linting, and general code quality, and counting a submission complete only if it showed none of them.

The note frames this as a gap between "algorithmic scoring" and holistic evaluation. Many benchmarks, including SWE-Bench Verified, score agents by whether they pass reusable automated test cases, because that scoring "is easy," runs fast, and needs no human review — but "many goals are difficult to represent with algorithmic scoring functions," among them documentation quality, code style, and mergeability. Rein notes that the popularity of reinforcement learning with verifiable rewards compounds the risk, since it optimizes agents specifically toward tasks that can be automatically scored, so "we might expect that AI systems will perform better on tasks that can be automatically evaluated compared to ones that can't." METR estimates that bringing the 15 manually reviewed PRs to a mergeable state would take a human an average of 42 minutes, falling to an average of 26 minutes (minimum 15 minutes) when restricted to the PRs that already passed the human-written tests — against an average 1.3 hours the original maintainers spent on these same issues, so even a test-passing agent PR leaves roughly a third of the task's human time still undone.

The note offers this as a candidate explanation for a puzzle elsewhere in the library: METR's own measured time horizon for Claude 3.7 Sonnet is about one hour, yet Do AI coding tools actually speed up experienced developers? found the same family of tools slowed, rather than sped, experienced developers on real tasks of comparable length. Rein's hypothesis is that the time-horizon figure is itself inflated by algorithmic scoring, since time-horizon and R&D benchmarks like the one behind When do AI agents outperform human research experts? also depend on scoring functions that can miss the mergeability, documentation, and code-quality failures this note measures directly. It also extends What shortcuts do high-accuracy agents actually take during benchmarks? and Does a single benchmark score actually predict agent readiness?, both of which argue that a pass-rate score can hide behavior that matters for deployment: this note supplies a specific, measured version of that gap — a 38 percent algorithmic pass rate against a 0 percent holistic-mergeability rate on the same tasks — rather than a general argument for why such a gap might exist.

The sample is small: 18 issues, 15 manually reviewed, from two repositories, one run per task manually scored, and the "time to fix" figures are estimates made by comparing agent PRs to the reference human PR rather than scores from the original PR authors. The note does not establish that algorithmic scoring overestimates performance on every benchmark or task type, only that it did here, for coding issues with unit tests, under this agent scaffold. The implication the evidence supports is narrower but still consequential: wherever a benchmark's pass/fail signal depends on tests that the task's real stakeholders did not write, a high score is weaker evidence of real-world usefulness than it appears.