Should benchmarks report token counts instead of frozen costs?

Agent benchmarks conflate two different jobs, the paper argues, and that conflation is specifically what breaks cost reporting. "AI evaluations are used by model developers and AI researchers to identify which changes to the training data and architecture improve accuracy (model evaluation) and also by downstream developers to decide which AI systems to use in their products for procurement decisions (downstream evaluation). The difference between model evaluation and downstream evaluation is underappreciated. This has led to much confusion about how to factor in the cost of running AI." For downstream evaluation specifically, it states, "cost is the actual construct of interest."

The mechanism the paper gives is that inference prices are not stable, so a cost figure frozen at benchmark-publication time becomes misleading for a downstream buyer: "inference costs do come down over time, and that greatly matters to downstream developers. It is unnecessary and counterproductive for the evaluation to stay frozen in time." Its prescription is to report the ingredients of cost rather than a single dollar figure — "downstream evaluations of agents should include input/output token counts in addition to dollar costs, so that anyone looking at the evaluation in the future can instantly recalculate the cost using current prices" — and it describes having "prototyped a simple example of such an interface" that lets users adjust token prices for their provider of choice. The paper illustrates the failure mode with NovelQA, a benchmark whose novels range "from 50,000 to over a million words" and which ranks submissions on a public leaderboard: fine for model evaluation, but "it would be misleading if used for downstream evaluation by a developer looking to build a bot for answering questions about novels," since the leaderboard "does not measure cost" and, because of how it poses questions, is not easy to retrofit with one.

This complements the companion note from this same source, Why do accuracy-only benchmarks hide the true cost of agents?, which argues cost belongs on the optimization axis alongside accuracy; this note narrows that argument to who the cost figure is for and why it goes stale. It also gives a concrete instance of the pattern this paper asks for in When do AI agents outperform human research experts?, which already reports agents as "roughly 15x cheaper per run" rather than a single dollar amount — closer to the unit-cost reporting this paper asks benchmarks to adopt as standard. And it supplies a reason behind the economics claimed in Can small language models handle most agent tasks?: a cost advantage stated in dollars ("10-30x cheaper SLM inference") is exactly the kind of figure this paper says goes stale without the underlying token counts.

The excerpt supports this distinction with one worked case (NovelQA) rather than a survey of how widely current agent benchmarks fail to separate model from downstream evaluation, so it does not establish how common the conflation is across the "over a dozen agent benchmarks" the paper references elsewhere. Nor does it demonstrate that its prototyped cost-adjustable interface actually corrects downstream decisions in practice. The implication the evidence supports is that any procurement decision drawn from a benchmark reporting only a dollar cost, with no token counts, should be treated as time-bound and potentially already out of date, rather than as a stable ranking.