Research Update: Towards Reconciling Slowdown with Time Horizons (algorithmic vs holistic evaluation)

Source: David Rein (METR); METR research note · 2025

On 18 real tasks from two large open-source repositories, early-2025 AI agents often implement functionally correct code that cannot be easily used as-is, because of issues with test coverage, formatting/linting, or general code quality.

This suggests that automatic scoring used by many benchmarks1 may overestimate AI agent real-world performance.

Many AI benchmarks use algorithmic scoring2 to evaluate how well AI systems perform on some set of tasks. For example, the popular benchmark SWE-Bench Verified measures whether an AI system passes test cases implemented in code by the original human PR authors. Algorithmic scoring makes it easy to evaluate a new system on the benchmark, because the tests are re-usable, run quickly, and do not require human intervention/manual review.

However, many goals are difficult to represent with algorithmic scoring functions. For example, judging the quality of documentation is difficult to do automatically—you can’t (easily) write test cases that evaluate this, because documentation is typically unstructured natural language.

Particularly given the recent popularity of methods like reinforcement learning with verifiable rewards, which relies on algorithmic scoring functions, we might expect that AI systems will perform better on tasks that can be automatically evaluated compared to ones that can’t.

If AIs are much more capable on tasks that can be automatically evaluated, we might overestimate how useful they are in the field, because we often task AI systems with work that can’t be automatically evaluated. We hypothesize that this contributes to the apparent gap between our measured time horizon of Claude 3.7 Sonnet of about one hour, and the slowdown effect we observe from our recent developer productivity RCT (which includes many tasks that take humans an hour to complete).

We took 18 issues from two repositories that were part of our recent RCT measuring the impact of AI on experienced open-source developer productivity. 15 of the tasks come from the stdlib-js repository, and 3 are from the hypothesis repository—they are both large repositories, with 8M and 100k lines of code respectively, and the 18 issues took human maintainers between 20 minutes and four hours to complete (averaging 1.3 hours).3 To select these issues, we filter to issues that have unit tests implemented by the original PR author, in increasing order of length for humans. Qualitatively, these repositories and issues are representative of the broader distribution of repositories involved in the RCT. We include several task examples in the appendix.

To give agents a fair shot at autonomously completing the tasks, we implement and test development environments for each task, using the base commit of the original human PR. This gives the agent a comparable setting to human repository maintainers, so they don’t need to (for example) install dependencies themselves, and can immediately start working on each task.

We have an Inspect ReAct agent using Claude 3.7 Sonnet attempt these PRs. We then took tests implemented by the human’s PRs and evaluated agents’ solutions against these. Specifically, we measure whether the agent’s solution passes all test cases implemented in the human’s original reference PR. We run these tests on the agent’s code directly in the task environment, to reduce the likelihood of dependency/environment discrepancies.

The agent is not shown the test cases, which makes it much less likely that they can reward hack—broadly, the agent is instructed to fully implement the given issue. We overall spend a few hours iterating on the agent’s instructions, primarily iterating on instructions related to testing their solutions and using version control properly to submit their implementations. The full prompt is available in the appendix.

We then randomly sample one run per task4, and manually evaluate the agent’s submitted PR. The goal of this manual evaluation/scoring is to holistically judge whether the agent’s PR would be mergeable in the repository without needing meaningful/substantial extra work. We then compare these manual scores to agents’ scores on the reference test cases described above.

We give each agent PR a binary 0 or 1 score on each of these axes, and consider a submission complete if it contains none of the failure modes (i.e. it’s mergeable as-is).

Claude 3.7 Sonnet in the Inspect ReAct agent scaffold has an average success rate of 38% (±19%, 95% CI) on these issues, as measured by the maintainer-written test cases, indicating that this agent often implements the core functional components of the issues correctly5. However, when manually reviewing a subset of these PRs6, none of them are mergeable as-is. This suggests that even when measuring AI capabilities at doing real software tasks, which is often considered the gold-standard method of evaluating real-world performance of AI systems in software development, algorithmic scoring based on unit tests overestimates model capabilities.

For each of the 15 PRs manually reviewed, we estimate how long we expect it to take a human to modify/fix the PR to be in a working/usable state. We estimate that on average, the agent PRs would take 42 minutes to modify/fix to be in a mergeable state. When filtering to runs where the agent passed the human-written test cases, we estimate that these PRs would take on average 26 minutes to fix (with a minimum time of 15 minutes). So, even when the agent implements the core algorithmic functionality correctly (i.e. it passes all of the human-written test cases), there is still a substantial amount of work needed for the PRs to be mergeable—the average time these PRs took the original maintainer is 1.3 hours, so the “time to fix” represents about a third of the total time needed for these PRs. Note that these estimates are made by comparing the agent’s PRs to the reference human PR, and are not done by the original PR authors.