Why do accuracy-only benchmarks hide the true cost of agents?

Agent benchmarking that scores only accuracy, the paper argues, misses the cost side of agent design, and this has driven today's agents toward needless complexity. The excerpt states "there is a narrow focus on accuracy without attention to other metrics. As a result, SOTA agents are needlessly complex and costly, and the community has reached mistaken conclusions about the sources of accuracy gains." It adds, plainly, that "useful agent evaluations must control for cost — even if we ultimately don't care about cost and only about identifying innovative agent designs," because "accuracy alone cannot identify progress because it can be improved by scientifically meaningless methods such as retrying."

The paper's proposed fix is to treat cost and accuracy as two axes of one design problem rather than reporting accuracy alone. It splits an agent's total cost into a one-time "fixed cost" (tuning hyperparameters — prompts, temperature, few-shot examples — for a task) and a per-run "variable cost" (tokens consumed each time the agent runs), noting "the more an agent is used, the more the variable cost dominates." Visualizing accuracy against cost as a Pareto frontier, it argues, "opens up a new space for agent design: jointly optimizing cost and accuracy, which can lead to agents that cost less while maintaining accuracy," by spending more upfront on design search (e.g., finding shorter prompts) to cut the recurring per-run cost. As a demonstration, on HotPotQA tasks the paper reports that jointly optimized Llama-3-70B and GPT-3.5 agents both become cheaper in total cost than the default DSPy implementation "after 1,350 tasks" — the fixed cost of optimization pays for itself once an agent runs often enough.

This gives cost a concrete place among the evaluation axes that Does a single benchmark score actually predict agent readiness? argues benchmarks systematically omit — where that note catalogs axes like privacy compliance and long-horizon retention, this paper names cost (and its fixed/variable split) as one such axis and shows a method for optimizing it alongside accuracy. It also sharpens the economic argument in Can small language models handle most agent tasks?: where that note argues SLMs are cheaper by default, this paper shows accuracy-only benchmarking is itself what obscures the cost savings available even within a single model class. It also complements When do AI agents outperform human research experts?, which already reports agents as roughly 15x cheaper per run than humans — exactly the kind of cost figure this paper says benchmarks should report alongside accuracy by default, not as an afterthought.

The excerpt demonstrates its cost-accuracy optimization only on HotPotQA against a DSPy baseline, with a break-even point ("after 1,350 tasks") specific to that task and those two models; it does not establish how often, or by how much, joint optimization reduces cost across other task types, agent architectures, or longer-horizon settings. The paper is also explicit that providers' prices shift unpredictably ("the cost of an API call might change overnight"), so even its own cost figures are time-bound. The implication the evidence supports is narrower than "agents can always be made cheaper without losing accuracy" — it is that benchmarks silent on cost cannot distinguish genuine efficiency gains from agents that are merely more expensive at the same accuracy, and that distinction requires deliberately measuring and reporting both.