How much does agent design choice actually cost?

The Holistic Agent Leaderboard (HAL) runs "21,730 agent rollouts across 9 models and 9 benchmarks in coding, web navigation, science, and customer service with a total cost of about $40,000," tracking dollar and token cost alongside accuracy for every run rather than accuracy alone. The paper's central cost finding is that "the Pareto frontier of accuracy and cost is often steep": "In only 1 of 9 benchmarks do we observe the most costly model run on the Pareto frontier," despite "the most expensive model costing an order of magnitude more than mid-tier options: Claude Opus 4.1 costs $15/$75 per million tokens while GPT-5 costs $1.25/$10." The frontier is also "usually sparse": "less than one-third of tested models are on the frontier for a given benchmark," with Gemini 2.0 Flash (7 of 9 benchmarks), GPT-5 (4 of 9), and o4-mini Low (4 of 9) — models differing in cost by an order of magnitude — most frequently occupying it.

Scaffold choice (the prompts, tools, and control logic wrapping a model) interacts with cost as strongly as model choice does: "on Online Mind2Web, SeeAct with GPT-5 Medium costs $171 while Browser-Use with Claude Sonnet 4 costs $1,577: a 9x difference in cost despite just a two-percentage-point difference in accuracy," and "Claude models perform better with BrowserUse, while OpenAI models achieve higher accuracy with SeeAct" — so "optimal agent design requires carefully matching models to scaffolds," not picking a model in isolation. Reasoning effort frequently fails to pay for itself too: across reasoning-effort comparisons on four models, "in 21 of 36 model-agent-benchmark combinations, increased reasoning effort produces equal or lower accuracy," and on 6 of 9 benchmarks token usage correlates positively with accuracy without a matching efficiency gain — more tokens buy accuracy inconsistently.

This adds a cost axis to Does a single benchmark score actually predict agent readiness?, which argues capability decomposes across axes that single benchmarks miss: HAL's contribution is showing cost is one such axis, cheap to track once the harness instruments it automatically, and that ranking agents by accuracy alone can invert the practical ranking once cost is priced in. It also extends When do AI agents outperform human research experts?'s observation that cost and accuracy trade off in ways a single benchmark score hides, though RE-Bench prices agents against human labor cost while HAL prices agents against each other and against scaffold choice. It pairs with the sibling finding in What shortcuts do high-accuracy agents actually take during benchmarks?, which shows accuracy and cost together still miss behavior that only log inspection exposes.

The excerpt does not say whether these cost rankings hold outside the four domains tested (coding, web navigation, science, customer service), or whether frontier pricing — which changes faster than benchmark cycles — would reorder the frontier by the time a reader checks it; the $40,000 total and per-model prices are a point-in-time snapshot from the paper's evaluation window, not a standing rate. It also does not separate how much of the "steep and sparse" pattern reflects genuine capability-cost tradeoffs versus provider pricing strategy. The finding supports choosing agents by measured cost-accuracy position on a benchmark close to the deployment task, not a general claim that cheaper models are better.