Measuring AI Ability to Complete Long Software Tasks

Despite rapid progress on AI benchmarks, the real-world meaning of benchmark performance remains unclear. To quantify the capabilities of AI systems in terms of human capabilities, we propose a new metric: 50%-task-completion time horizon, the time humans typically take to complete tasks that AI models can complete with 50% success rate. We first timed humans with relevant domain expertise on a combination of RE-Bench, HCAST, and 66 novel shorter tasks. On these tasks, current frontier AI models such as o3 have a 50% time horizon of around 110 minutes. Furthermore, frontier AI time horizon has doubled approximately every seven months since 2019, though the trend may have accelerated since 2024. The increase in AI models’ time horizons seems to be primarily driven by greater reliability, ability to adapt to mistakes, logical reasoning, and capacity for tool use. We discuss the limitations of our results—including their degree of external validity—and the implications of increased autonomy for dangerous capabilities. If these results generalize to real-world software tasks, extrapolation of this trend predicts that within 5 years, AI systems will be capable of automating many software tasks that currently take humans a month.

Introduction. In the last five years, frontier AI systems have undergone a dramatic transformation in capabilities, evolving from basic text generation [1] to autonomously executing complex multi-hour machine learning research projects [2]. Sufficiently capable AIs could perform dangerous, highly complex actions like autonomous development of chemical, biological, radiological or nuclear weapons (CBRN) and self-replication and adaptation outside human control [3]. Understanding AI capabilities helps inform the development of safety guardrails as systems become increasingly powerful. In particular, many frontier AI developers have committed to using measures of specific AI capabilities to determine the necessary risk mitigations for their frontier AI systems. Robust benchmarks that can accurately track and forecast AI capabilities thus form the foundation for responsible AI governance and risk mitigation.

However, existing benchmarks face several key limitations. First, they often consist of artificial rather than economically valuable tasks. Second, benchmarks are often adversarially selected for tasks that current models struggle with compared to humans,1 biasing the comparison to human performance. Most critically, individual benchmarks saturate increasingly quickly [6], and we lack a more general, intuitive, and quantitative way to compare between different benchmarks,2 which prevents meaningful comparison between models of vastly different capabilities (e.g., GPT-2 versus Claude 3.7 ). As a consequence, while the last few years have seen dramatic increases in AI performance on many individual benchmarks, understanding the progress of AI capabilities in general has required estimating the qualitative difficulty of the latest benchmarks AI systems can pass.

We propose tracking AI progress over time using the task completion time horizon: the duration of tasks that models can complete at a certain success probability, providing an intuitive measure of real-world capability compared to humans. As models may not reliably complete all tasks of a given length, we operationalize this by measuring the X%-(task completion) time horizon–the length of tasks that models can complete approximately X% of the time.

We prototype this methodology using three datasets designed to capture skills required for research or software engineering (Section 2.1), totaling 170 tasks with a wide range of difficulty: HCAST [8], RE-Bench [2], and Software Atomic Actions (SWAA), a new suite of shorter software tasks that can measure pre-2023 models (Appendix B.1.3). Using skilled human baseliners, we estimate the duration that a domain knowledgeable human (without task-specific context) takes to complete these tasks (Section 2.2). We evaluate the performance of 12 frontier models from 2019 to 2025 on these tasks (Section 2.3). Using methodology inspired by human psychometric studies, we then estimate the duration of tasks that models can complete with 50% success rate—the 50% time horizon (Section 3.1).

We find that the 50% time horizon has been growing exponentially from 2019–2025 on these tasks, with a doubling time of approximately seven months (Figure 1). We compare our main result with our exploratory results on non-SWAA tasks, finding that the 2023–2025 horizon growth rate is about 20% faster than the 2019–2025 rate (Section F.1). We also measure the 80% time horizon of models

Related work. Here, we briefly discuss related work. See Appendix A for an expanded discussion.

Agentic AI capability benchmarks Many recent benchmarks aim to measure the capability of AI systems to act as agents in general, including AgentBench [9], ToolBench [10], GAIA [11], TheAgentCompany [12], the Berkeley Function Calling Leaderboard [13]. In this work, we study AI agent performance on RE-Bench[2], HCAST [8], and SWE-Bench Verified [7], which focus on machine learning and software engineering tasks. Other similar benchmarks include MLAgent- Bench [14], MLEBench [15], and DSBench [16]. In concurrent work, Miserendino et al. [17] create the SWE-Lancer benchmark, consisting of 1400 freelance software engineering tasks gathered from Upwork with real-world payouts totalling above $1 million.

AI capability forecasting This work builds on previous efforts to forecast the capabilities of AI systems over time, including efforts to contextualize benchmark performance using human experts Phuong et al. [3]; Murray et al. [18] and efforts to forecast performance on particular benchmarks Owen [19]; Pimpale et al. [20]. Contextualizing AI capabilities using human time horizon was proposed by Ngo [21], and similar concepts were discussed in Carlsmith [22] and Cotra [23].

Human psychometric methods Our methodology draws inspiration from human psychometric testing, especially Item Response Theory [24; 25; 26].

Method. Our tasks comprise three distinct task suites:

1. A subset of HCAST [8]: 97 diverse software tasks ranging from 1 minute to 30 hours.3 2. RE-Bench [2]: 7 difficult ML research engineering tasks, all eight hours long.

2. Software atomic actions (SWAA): 66 single-step tasks representing short segments of work by software developers, ranging from 1 second to 30 seconds.

All tasks are automatically scored with a continuous score or binary threshold; details of how we normalize and process scores are given in Section 3.1, and other task suite details in Appendix B.

Example tasks See Table 1 for five example tasks. Tasks under 1 minute measure knowledge relevant to professional software engineering. At 10 minutes, tasks can include the easiest meaningful step of a real software project (e.g. configuring a common open-source package). The shortest tasks that could be standalone economically relevant projects take around one hour; by eight hours tasks represent valuable software projects. Many more examples of these tasks are available in Wijk et al. [2] and Rein et al. [8].

In order to ground AI agent performance, we also measure the performance of multiple human “baseliners” on most tasks and recorded the duration of their attempts. In total, we use over 800 baselines totaling 2,529 hours, of which 558 baselines (286 successful) come from HCAST and RE- Bench, and 249 (236 successful) from the shorter SWAA tasks. 148 of the 169 tasks have human baselines, but we rely on researcher estimates for 21 tasks in HCAST.

Our baseliners are skilled professionals in software engineering, machine learning, and cybersecurity, with the majority having attended world top-100 universities. They have an average of about 5 years of relevant experience, with software engineering baseliners having more experience than ML or cybersecurity baseliners. For more details about baselines, see Appendix C.1.

To convert agent run data to task completion time horizon, we first convert the agent performance on each task to a binary value (success or failure). Many tasks are naturally binary, including all SWAA tasks. Continuously scored tasks are binarized via a task-specific threshold chosen to represent human performance. For HCAST, the task-specific threshold is the same “target score” the human baseliner tries to achieve, which we also use to filter for successful runs. RE-Bench tasks have a fixed time rating of 8 hours, so the task-specific threshold is the average score of 7-9–hour human runs.

Once we have agent success rates and time ratings for each task, we fit time horizons by performing the following logistic regression:5 where ttask is the geometric mean time of successful human baselines, and hagent and βagent are learned parameters, with hagent representing the 50% time horizon.

Discussion. Finally, we attempt to extrapolate the trend on these tasks to one-month (167 hours) AI (Section 5), finding that if the trend continues and observed performance trends generalize to real-world tasks, an 80% confidence interval for the release date of AI that can complete 1-month long software tasks spans from mid-2028 to mid-2030 (Section 5)– or even as soon as early 2027 if the 2024–2025 trend continues.

Conclusion. Summary In this paper, we proposed an intuitive, quantitative metric for AI capabilities: the task completion time horizon, which relates AI performance on tasks to the typical length of time human experts require to complete the tasks. We constructed a dataset of 66 shorter SWAA tasks, combined these with existing benchmarks, and conducted baselines to estimate their difficulty. To measure the trend in time horizon, we benchmarked 11 frontier AI models released between 2019 and 2025 on this dataset, and found (Section 3) that the 50% task completion time horizon on these tasks has been growing exponentially from 2019–2025 with a doubling time of approximately seven months (Figure 1). We also noted important limitations of current systems, particularly their lower performance on less structured, “messier” tasks (Appendix F.2).

Limitations. Task distribution limitations We note several differences between this task suite and realistic tasks in Appendix B.2: in short, realistic tasks rarely have automatic scoring, and often require dealing with other agents, resource constraints, dynamic environments, and high reliability requirements. Such differences cast doubt on whether the rapid performance improvements seen on this task suite (and other benchmarks) will generalize. It may be that benchmarks like HCAST and SWE- Bench Verified are insufficient for forecasting real-world AI capabilities, and accurate forecasting will require deliberate construction of more realistic benchmarks without these limitations.

Future changes in time horizon trends The time horizon trend may significantly change in the future due to factors including agency training, compute scaling, and automation of AI research and development. Agency training and eventual automation of AI R&D are more likely to speed up the trend, whereas compute scaling could slow it (but be partially offset by algorithmic improvements); a future artificial general intelligence (AGI) could technically have an infinite time horizon. See Appendix E for details.