Is benchmark progress outpacing the cost to achieve it?

Combining historical Artificial Analysis token-price snapshots (collected via the Internet Archive, April 2024–November 2025) with Epoch AI's Benchmark Hub, a regression analysis finds that "the price for a given level of benchmark performance has decreased remarkably fast, around 5× to 10× per year, for frontier models on knowledge, reasoning, math, and software engineering benchmarks" (GPQA-Diamond, AIME, SWE-bench Verified). At the same moment, "the price of running frontier models is rising between 3× to 18× per year due to bigger models and larger reasoning demands." Both trends are measured over the same window and the same model population, not inferred from different sources.

The mechanism the paper gives for the apparent contradiction is that marginal frontier gains are increasingly bought with spending rather than delivered free by technical progress. For GPQA-Diamond specifically, "roughly half of measured progress is associated with increasing inference prices: holding benchmark cost fixed, performance improves at about half the rate implied by the unconditional time trend." The rate of price decline is also uneven across the performance range — models in the highest GPQA-Diamond bin declined in price by 31× per year, versus 1.7× per year in the lowest bin — which the paper attributes tentatively to heavier use of distillation or mixture-of-experts architectures at the high end.

This gives empirical weight to the claim in Do automated benchmarks hide what frontier AI systems can really do? that benchmark scores without their cost are uninterpretable: here the distortion is not just task-design bias toward gradable, short-horizon problems but a demonstrable spending effect, where a chunk of headline benchmark gains tracks post-training and RL compute spend rather than price-independent capability. How fast is AI inference actually becoming more efficient? decomposes the 5×–10× figure reported here into its separate drivers.

The paper is explicit that its 5×–10× estimate is lower than Cottier et al. (2025)'s reported 10–1,000× range, attributing the gap to its benchmark- rather than token-level approach, its narrower April 2024–November 2025 window, and denser model coverage — so the figure should not be read as a universal "inference is collapsing" rate. The SWE-bench Verified data is also described as "so limited that our confidence bounds are large enough to be consistent with there having been no progress at all," so the 5×–10× claim rests more heavily on GPQA-Diamond and AIME than on coding benchmarks. The practical implication the paper draws is that evaluators should publish and weigh benchmarking price alongside scores, since a model that raises a benchmark score mainly by consuming more compute represents a smaller real capability advance than the headline suggests.