Holistic Agent Leaderboard: The Missing Infrastructure for AI Agent Evaluation

AI agents have been developed for complex real-world tasks from coding to customer service. But AI agent evaluations suffer from many challenges that undermine our understanding of how well agents really work (Figure 1). We introduce the Holistic Agent Leaderboard (HAL) to address these challenges. We make three main contributions. First, we provide a standardized evaluation harness that orchestrates parallel evaluations across hundreds of VMs, reducing evaluation time from weeks to hours while eliminating common implementation bugs. Second, we conduct three-dimensional analysis spanning models, scaffolds, and benchmarks. We validate the harness by conducting 21,730 agent rollouts across 9 models and 9 benchmarks in coding, web navigation, science, and customer service with a total cost of about $40,000. Our analysis reveals surprising insights, such as higher reasoning effort reducing accuracy in the majority of runs. Third, we use LLM-aided log inspection to uncover previously unreported behaviors, such as searching for the benchmark on HuggingFace instead of solving a task, or misusing credit cards in flight booking tasks. We share all agent logs, comprising 2.5B tokens of language model calls, to incentivize further research into agent behavior. By standardizing how the field evaluates agents and addressing common pitfalls in agent evaluation, we hope to shift the focus from agents that ace benchmarks to agents that work reliably in the real world.

Introduction. AI agents are being developed across domains from software engineering (Anthropic, 2025) to web task automation (OpenAI, 2025a). AI agent benchmarks have also proliferated (Zhou et al., 2024; Xie et al., 2024; Abhyankar et al., 2025; Deng et al., 2023; Zhang et al., 2024; Huang et al., 2024; Zhu et al., 2025b). Yet there are many challenges for proper evaluation (Figure 1). First, evaluation infrastructure is non-standardized, prohibitively slow, and error-prone. A single benchmark could take weeks to run serially, and as a result, leaderboards are often not updated with the latest models (Challenges #1-#3. Second, agents vary widely in costs, but evaluations rarely report these costs. Scaffolds dramatically impact both accuracy and cost, yet comparisons across scaffolds are rare (Challenges #4-#6). Third, agents can exploit shortcuts that inflate benchmark scores and take actions that would be catastrophic in deployment. Yet, current evaluations rarely detect or penalize such behavior (Challenges #7-#8).

To address these shortcomings, we develop HAL, the Holistic Agent Leaderboard. It is a unified evaluation framework for reproducible, cost-controlled agent benchmarking with automated agent log analysis. Existing evaluation frameworks like HELM (Liang et al., 2023) and LM Evaluation Harness (Biderman et al., 2024) have standardized language model benchmarking, but agent evaluation poses fundamentally different challenges. While LLMs respond to prompts with text, agents navigate complex environments over extended time horizons, using tools from browsers to bash shells, often consuming hundreds of thousands of tokens per rollout. They can fail catastrophically or get trapped in loops in ways that simple text generation cannot. These unique characteristics demand purpose-built infrastructure that tracks not just what agents output, but how they achieve it, what they cost, and where they break. HAL provides this infrastructure.

HAL contains three main components:

1. Unified evaluation harness with distributed orchestration: We develop an open source harness that standardizes agent evaluation across diverse benchmarks. The harness is frameworkagnostic, requiring light-weight modifications to integrate scaffolds, while automatically instrumenting cost tracking, logging all API calls, and capturing complete execution traces (Section 2). Through orchestration across hundreds of virtual machines, we reduce evaluation time from weeks to hours (Challenge #1). The system handles heterogeneous execution environments, from web browsers to code repositories (Challenge #2), through a unified interface that researchers can invoke with a single command to update leaderboards (Challenge #3).

2. Multidimensional leaderboard with large-scale evaluation results: We conduct 21,730 agent rollouts spanning 9 benchmarks and 9 models, costing about $40,000 in compute (Sections 3 and 4.1). Our leaderboard tracks performance across three dimensions: agent scaffolds, models, and benchmarks, revealing previously hidden interactions between these factors (Challenge #6). While some prior work develops leaderboards for a single domain (e.g., web agent evaluation (Chezelles et al., 2025) or tool use (Bhavsar & Bronsdon, 2025)), we implement standardized evaluation for 9 challenging agent benchmarks across domains (Challenge #5), spanning web navigation (Online Mind2Web, AssistantBench, GAIA), coding (SWE-bench Verified Mini, US- ACO), scientific research (CORE-Bench Hard, ScienceAgentBench, Scicode), and customer service (TAU-bench Airline). Unlike existing leaderboards, which typically focus on accuracy (Liu et al., 2023; Ma et al., 2024; Xi et al., 2024; UK AISI, 2025), we present Pareto frontiers of accuracy versus cost (both dollar cost and token cost), enabling practitioners to select agents based on real-world constraints (Challenge #4). 3. Automated analysis of agent logs: We collect over 2.5 billion tokens of language model calls from our 21,730 agent rollouts. We conduct LLM-aided analysis of logs for four benchmarks (CORE-Bench, AssistantBench, SciCode, TAU-bench Airline) using Docent (Meng et al., 2025). Our findings show that analysis of agent logs is an essential component of conducting agent evaluations (Section 4.2). In particular, we find that (i) many agents take shortcuts such as looking up the gold answer for a task by looking up the benchmark on HuggingFace rather than actually solving the task of interest (Challenge #7); (ii) agents often take actions that would be catastrophic if deployed to real-world products, such as using a wrong credit card to make flight bookings (Challenge #8); (iii) log analysis can help uncover bugs and other shortcomings in agent scaffolds. For example, we uncovered a major bug in a scaffold for the TAU-Bench benchmark, after we had conducted evaluations that cost a significant amount to run.*; and (iv) log analysis can help uncover potential improvements to benchmark design.

Related work. HAL is an ongoing project that we plan to maintain as a community resource. We extensively discuss related work on standardizing evaluation and research infrastructure in Section A1. Over the next two years, we plan to maintain and improve HAL in various ways, such as by continuing to add challenging benchmarks that correspond to real-world tasks, running evaluations with updated models, developing stronger scaffolds, and carrying out large-scale automated log analysis of all results.

Method. We developed the HAL harness to provide the infrastructure needed for reproducible, cost-controlled agent evaluation at scale (Table 2). At its core, the harness accepts any agent that exposes a minimal Python API and orchestrates its evaluation across diverse benchmarks, from web navigation tasks to code repositories. The harness decouples scaffold implementation from benchmark execution. Through integration with Weave (Wandb.ai, 2024) for comprehensive logging, LiteLLM (BerriAI, 2025) for cross-model compatibility, and support for many execution environments (local, Docker, and Azure VMs), the harness drastically reduces the time it takes to evaluate agents. The system automatically tracks token usage and costs, enabling systematic cost-aware analysis previously missing from agent evaluation. This infrastructure makes possible the comprehensive evaluations we present in Sections 3 and 4. We provide additional implementation details and a system diagram for the harness in Section A2.

We evaluate agents across three dimensions: models, benchmarks, and agent scaffolds. This threeaxis design reveals interactions invisible to traditional single-benchmark evaluations. We select 9 benchmarks spanning four domains (coding, web navigation, scientific research, and customer service), 9 models, including reasoning and non-reasoning models, and multiple agent scaffolds. We Benchmarks. We identified four major domains where agents are being developed and deployed: web navigation, software engineering, scientific research, and customer service. For each domain, we selected benchmarks that evaluate agent capabilities and represent tasks that had high construct validity (Zhu et al., 2025a). The 9 benchmarks we selected are listed in Table 3. † They test whether agents can navigate complex interfaces, write correct code, conduct scientific analysis, and handle multi-turn interactions. Note that the list of domains is not exhaustive; there are many other domains that we do not focus on for this version of HAL, such as cybersecurity and retail. While this is outside the scope of the current paper, we plan to increase the scope of our evaluations in future iterations of HAL.

Models. We evaluated 9 models (Table A11) that span the spectrum of capabilities and costs, including frontier models (OpenAI’s o3, GPT-4.1, GPT-5, and Anthropic’s Claude Sonnet 3.7 and Opus 4.1) as well as cost-efficient models that balance performance with affordability (DeepSeek V3, DeepSeek R1, o4-mini, Gemini 2.0 Flash). We also evaluated some models with different reasoning efforts to study the cost-accuracy tradeoffs of inference-time compute (Claude Sonnet 3.7 Agent scaffolds. Agent scaffolds comprise the prompts, tools, and control logic that allow language models to solve agentic tasks (Kapoor et al., 2025). We used task-specific scaffolds from prior work for each benchmark except TAU-Bench. (The TAU-bench Few Shot agent suffered from data leakage that invalidated our results; we discovered this through automated log analysis and excluded this scaffold from our analysis. See Section 4.2 and Appendix A5 for details.) These scaffolds leverage domain knowledge and specialized tools: CORE-Agent (Siegel et al., 2024) has prompts to specifically direct it to solve reproducibility tasks in CORE-Bench, SWE-Agent (Yang et al., 2024) provides a custom shell interface for code editing, and SAB Self-Debug (Chen et al., 2025) implements iterative refinement for scientific tasks. In addition, we develop a “generalist” scaffold that works across multiple benchmarks, and evaluate three benchmarks (CORE-Bench Hard, TAU-bench Airline, SWE-bench Verified Mini) using this scaffold, to understand the tradeoffs between special- Automated log analysis Beyond the analysis of accuracy and cost on benchmarks, we systematically analyzed agent logs to identify shortcuts and reliability failures that traditional metrics miss (Challenges #7-#8). We used Docent (Meng et al., 2025) to analyze AI agent transcripts at scale and to examine patterns across execution traces generated by our evaluations.

Discussion. We present two complementary types of analysis from our large-scale agent evaluation. First, we examine multidimensional benchmark results (Section 4.1) that reveal patterns in accuracy, cost, and token usage across models, benchmarks, and agent scaffolds. Many of our insights challenge common assumptions about model performance. For example, we find that higher reasoning effort doesn’t always improve accuracy (Figure 3).

Our evaluation captures both accuracy and cost for each agent run, enabling analysis beyond traditional single-metric leaderboards. In particular, we find:

1. The Pareto frontier of accuracy and cost is often steep. In only 1 of 9 benchmarks do we observe the most costly model run on the Pareto frontier. This is despite the most expensive model costing an order of magnitude more than mid-tier options: Claude Opus 4.1 costs $15/$75 per million tokens while GPT-5 costs $1.25/$10. [Figure 2] 2. The Pareto frontier of accuracy and cost is also usually sparse. On average, less than one-third of tested models are on the frontier for a given benchmark. The three models most frequently on the Pareto frontier, Gemini 2.0 Flash (7 of 9 benchmarks), GPT-5 (4 of 9), and o4-mini Low (4 of 9) differ in cost by an order of magnitude. [Figure 2] 3. Improvements to accuracy on agentic benchmarks are usually not accompanied by greater token efficiency. On 6 of 9 benchmarks, there is a positive correlation between token usage and accuracy. This is somewhat intuitive given the rise of inference-time scaling, but indicates this scaling has not yet yielded dramatic efficiency gains in long time horizon tasks. [Figure A2] 5. The effectiveness of greater test-time compute on accuracy is inconsistent across benchmarks. When comparing four models where we have reasoning comparisons (Claude Opus 4.1, Claude Sonnet 4, Claude-3.7 Sonnet, and o4-mini), we observe that in 21 of 36 model-agentbenchmark combinations, increased reasoning effort produces equal or lower accuracy. More reasoning does not always mean better results. [Figure 3] 6. Agent scaffolds create drastic differences in cost and accuracy. The choice of scaffold can be consequential in determining the final cost and accuracy of the agent. For example, on Online Mind2Web, SeeAct with GPT-5 Medium costs $171 while Browser-Use with Claude Sonnet 4 costs $1,577: a 9x difference in cost despite just a two-percentage-point difference in accuracy. Model-scaffold interactions are also complex. Claude models perform better with BrowserUse, while OpenAI models achieve higher accuracy with SeeAct. These patterns suggest that optimal agent design requires carefully matching models to scaffolds. [Figure A3c] 7. Generalist scaffolds sacrifice substantial accuracy for cross-benchmark compatibility. In cases where the same models are used for both task-specific and generalist scaffolds across three benchmarks, task-specific agents consistently outperform. On CORE-Bench Hard, the task-specific CORE-Agent outperforms the generalist scaffold on 9 of 12 runs. A similar gap appears on SWE-bench Verified Mini (11 of 12). The generalist scaffolds also cost less in 20 of 24 model comparisons, but the accuracy penalty is substantial. [Figures A3a, A3b] 4.2 RESULTS FROM THE AUTOMATED ANALYSIS OF AGENT LOGS 1. Agents often take shortcuts. Agents can succeed at tasks by correctly solving them or by taking shortcuts and guessing. On AssistantBench, we observe that some agents can achieve higher accuracy by guessing from common patterns rather than following the required search process. There were also eight cases where agents found a gold answer, either by finding the answers of the dataset on HuggingFace or locating them on arXiv (Table A8).

Conclusion. The agent evaluation ecosystem suffers from fundamental infrastructure gaps that we try to address using HAL. We show that without standardized harnesses, multidimensional evaluation, and log analysis, the field cannot distinguish between genuine capability and benchmark gaming, nor can it assess economic viability for deployment.

More broadly, three changes are essential for progress in the AI agent evaluation ecosystem. First, systematic log analysis must become a necessary component of agent leaderboards. Our findings show that agents with identical accuracy scores can exhibit vastly different behaviors, with some agents taking shortcuts or highly costly actions such as using incorrect payment methods. Second, evaluation infrastructure must be standardized rather than repeatedly reimplemented, enabling fair comparisons across benchmarks, models, and scaffolds. Our experience reveals that this is easier said than done: providers swap model weights without notice, libraries break when new models don’t match hardcoded patterns, and providers can serve the same model with different quantizations over time (we document many such hurdles in Section A3). Third, evaluations must capture the full spectrum of performance dimensions, from token usage to failure modes to scaffold interactions. As companies increasingly aim to develop powerful agents, rigorous evaluation infrastructure like HAL will become more critical to evaluate their claims, helping ensure that agents work reliably in practice and not just in benchmarks. We plan to continue to invest significantly in maintaining and updating the project.

Limitations. While we tried to solve many of the challenges of AI agent evaluation, there are many other challenges we leave for future work. For example, in addition to dollar and token costs, users might want a sense of the latency of agents in solving a task. However, since we conduct massively parallel evaluations, the latency data we collect for solving a task might have additional variance due to issues such as rate-limit errors on APIs, which would be unrepresentative of real-world use. We document this and other limitations of HAL, as well as our plan for addressing them, in Section A4.