The Price of Progress: Price Performance and the Future of AI

Abstract Language models have seen enormous progress on advanced benchmarks in recent years, but much of this progress has only been possible by using more costly models. Benchmarks may therefore present a warped picture of progress in practical capabilities per dollar. To remedy this, we use data from Artificial Analysis and Epoch AI to form the largest dataset of current and historical prices to run benchmarks to date. We find that the price for a given level of benchmark performance has decreased remarkably fast, around 5× to 10× per year, for frontier models on knowledge, reasoning, math, and software engineering benchmarks. These reductions in the cost of AI inference are due to economic forces, hardware efficiency improvements, and algorithmic efficiency improvements. Isolating out open models to control for competition effects and dividing by hardware price declines, we estimate that algorithmic efficiency progress is around 3× per year. However, at the same time, the price of running frontier models is rising between 3× to 18× per year due to bigger models and larger reasoning demands. Finally, we recommend that evaluators both publicize and take into account the price of benchmarking as an essential part of measuring the real-world impact of AI.

Introduction. A critical dimension of the real-world impact of language models (and AI systems in general) is cost, yet it is often ignored in evaluation discourse. Recent discussion suggests large declines in the price of accessing high LLM performance (Appenzeller, 2024). At the upper end, Cottier et al.

In this study, we examine how the price to run LLMs changes for a given performance level and how performance changes with price. This sheds light on key drivers and important heterogeneity underlying trends in AI adoption and ability. We combine benchmark results from the Epoch AI Benchmark Hub (Epoch AI, 2024) with historical Artificial Analysis price snapshots collected via the Internet Archive, yielding the largest dataset of benchmark-level AI prices we know of (Appendix A). The dataset spans knowledge and reasoning (GPQA-Diamond; GPQA-D), mathematics (OTIS Mock AIME 2024–2025; AIME), and software engineering (SWE-bench Verified; SWE-V).

Using this data, we estimate regressions that recover an exponentially decreasing price trend conditional on performance—a proxy for inference efficiency progress. We use (i) pooled regressions with performance controls and (ii) within-performance-bin regressions; both approaches give consistent qualitative results.

We find overall progress closer to 10× than 1,000×, with estimates generally lower than Cottier et al. (2025) and closer to Erol et al. (2025). We attribute the gap to three factors: (1) a benchmarkrather than token-level approach that accounts for the rising token usage of recent (especially reasoning) models; (2) a focus on April 2024–November 2025, where we have high-quality price data, rather than earlier periods that may reflect unusually rapid early-stage change; and (3) substantially denser coverage, with nearly an order of magnitude more models per benchmark in our fits.

We also estimate separate trends for open-weight models. Because open-weight models can be run by anyone, they Price-performance improvements, however, do not tell the whole story. We also analyze how much benchmark progress is associated with higher inference spending rather than price-independent technical advances. For GPQA- Diamond this effect is particularly strong, roughly half of measured progress is associated with increasing inference prices: holding benchmark cost fixed, performance improves at about half the rate implied by the unconditional time trend. Relatedly, while per-token prices have generally declined, the cost of running frontier-level models has nonetheless risen approximately exponentially (about 3– 18× per year). This apparent paradox reflects the growing inference required for marginal gains at the frontier, with implications for both the pace of AI development—since evaluation costs may track post-training and reinforcement learning costs—and access to state-of-the-art capabilities.

In addition to establishing these findings we illustrate the current price-performance Pareto-frontier. This helps us identify the importance of open models and mixture of experts in the broader LLM ecosystem.

Finally, we argue that evaluations should be more transparent about computational resource usage. Without such data, it is difficult to distinguish meaningful technical improvements from progress driven primarily by greater compute expenditure.

Related work. (2025) finds that, controlling for benchmark performance, token prices may be falling by 10–1,000× per year, while Erol et al. (2025) report more moderate cost-of-pass declines on MATH 500 and AIME 2024 of 24.5× and 3.23× per year. Understanding these trends matters for forecasting when models become cost-competitive with labor and for democratizing access to state-of-the-art capabilities.1

Method. To measure the trend in benchmark price-performance controlling for quality, we run a variety of regressions on a dataset of benchmarking costs. We first describe our data collection before we turn to our regression results.

We collect data on input and output token prices over time by gathering Internet Archive data from Artificial Analysis. This data ranges from April 2024 to November 2025 and contains token-level price data from proprietary and open-weight model inference providers (OpenAI, Deepinfra, Cerebras, etc). We collect exclusively the lowest input and output token prices (the full price data across all providers is not available on the Internet Archive). We focus on the cheapest available provider to better control for the price/latency tradeoff and model the lower bound of model inference prices (see Erdil (2025)). In some cases, the lowest input and output prices were offered by different providers. However, in the vast majority of cases, the lowest input and output price are offered by the same provider, and when they are not, the resulting costs differences were negligible. In addition, we remove datapoints with 0 token cost as these are not reflective of actual inference prices.

We supplement our dataset with data from Epoch AI’s benchmarking hub (Epoch AI, 2024). This data includes models’ benchmark performance as well as the number of inputs, outputs, reasoning, and cached tokens used to run the benchmark. We compute the benchmark price by multiplying the corresponding tokens by their corresponding prices sourced from Artificial Analysis.

2.2. General Regression Approach Regression samples. In our main specification, we estimate the regression for models that are on the Pareto frontier of price-performance at a given time. Specifically, we filter out models that are Pareto dominated by earlier models that have a better benchmark score and a lower price to run a given benchmark. We call models in this group Pareto models. This yields a more relevant sample, as economicallyoptimizing users will typically choose the cheapest model for a given performance level. In additional analyses (see Table 1), we also run the regression for all available models and compare the results.

Finally, we also run separate regressions for open-weight models. Open-weight models can, in principle, be run and modified by anyone. We therefore expect that they will not be priced at a significant markup relative to the necessary GPU resources to run them. As a result, we argue that the trend in open-weight models more accurately measures technical progress rather than economic effects such as increased market pressure, and that the difference between our results for open-weight and proprietary models is (to some extent) informative about non-technological drivers of price changes, such as competition.

One potential concern is that open-weight models might not lie on the overall technical frontier. However, their performance appears to parallel frontier models with a lag of a few months to a year (Denain, 2024), so the general technical trend should be approximately similar.3 The results of our different fits are shown in Fig. 1.

Discussion. Nonetheless, the average reduction rate looks similar to our estimates for GPQA-Diamond and AIME. However, our estimates are much lower than those of Cottier et al. (2025). We attribute our lower estimates to three main factors. First, we use a benchmarkrather than a token-level approach, which compensates for the increased number of tokens used in recent models. This is particularly important for reasoning models, which can have a lower pertoken cost but a higher overall cost due to generating more tokens. Second, we examine the period from April 2024 to November 2025, where we have high-quality price data. Cottier et al. (2025) and related work examine price trends since April 2023 and late 2022, respectively. Therefore, our data do not capture possible price changes during the very initial stages of a new technology. Finally, we include almost an order of magnitude more models per benchmark in our fit than Cottier et al. (2025).

To get a better sense of the role of algorithmic progress (i.e., technical non-hardware progress) (Ho et al., 2024), we estimate β2 (the price decline) for open-weight models only. Additionally, we divide the estimated price trends by the decline in hardware prices as reported in Rahman (2024). After these adjustments, the remaining price reduction factor is around 3× per year, which can be interpreted as the contribution of algorithmic progress to declining price. Our estimates for algorithmic progress are more similar to experimental measures of energy efficiency gains like (Saad- Falcon et al., 2025) which find 3.1× gains from 2023-2025 controlling hardware. See Table 2 for more details on our measurement.

We also display our factor decomposition in Fig 2. We estimate the price performance decrease due to decreased competitive margins by dividing the rate of progress in closed-weight models by the rate of progress in open-weight models (which have much less monopolistic power). We see that our estimate of algorithmic progress is significantly stronger than hardware progress and our estimates of competitive price effects for the time period we measure from 2024-2025.

Similar to Cottier et al. (2025), we observe faster trends at the higher-quality frontier. For instance, in Fig. 3, models in the highest GPQA-Diamond bin declined in price by 31× per year, whereas models in the lowest bin declined by only 1.7× per year. This pattern may indicate that different economic or technical factors drive cost reductions in highercapability models (e.g., greater use of distillation or Mixtureof-Experts (MoE) architectures).

In addition, we find that the closed-weight model trend is slightly faster than the open-weight model trend. This is particularly pronounced for closed-weight models in the 40%– 60% group, where we see a sudden drop in price that is not mirrored in open-weight models, hinting at non-technical competitive effects. tion is that AIME and SWE-bench Verified have benefited significantly from RL enhancements, which don’t directly increase inference costs.

As shown in Fig 7, progress for models with bounded benchmark cost has plateaued for GPQA-Diamond. However, repeating the same analysis in Appendix D, we show that SWE-bench Verified and AIME have similar benchmark growth regardless of benchmark/inference budget.

AIME. When doing the same comparison for SWE-bench Verified and AIME, we see the frontier is progressing 12% and 14% faster than the cost-adjusted Pareto models, respectively.

Conclusion. At first glance, our analysis seems to point in different directions. The overall price for accessing a given level of LLM performance has dropped significantly, by 5× to 10× per year, although still substantially less than the reported 1000× upper bound by Cottier et al. (2025). However, like Cottier et al. (2025), we find much larger price declines for higher-performance models—almost 32× per year. For the least performant models, by contrast, we see much smaller price declines, around 1.7× per year, close to the estimates for energy efficiency improvements in AI models overall (Saad-Falcon et al., 2025). In terms of benchmarks, progress looks similar across GPQA-Diamond and AIME, but the data for SWE-bench Verified is so limited that our confidence bounds are large enough to be consistent with there having been no progress at all.

Despite these large price decreases, the benchmarking costs of models in our dataset have remained constant or increased as the demand for quality has risen, leading to larger models and longer reasoning. This is a particularly strong tension in frontier models: while a fixed capability level becomes dramatically cheaper over time, achieving frontier performance continues to demand higher frontier expenditure. Further, for some benchmarks a significant portion of apparent benchmark progress can be attributed to using more expensive models.

Taken together, these findings suggest that focusing on benchmark scores alone does not provide a full view of the nuanced nature of AI progress. A model that substantially improves benchmark performance but only by consuming dramatically more computational resources represents a The Price of Progress: Price-Performance and the Future of AI smaller advance in practical capability than headline scores suggest. Hence, we must take into account price if we want to understand progress.

We therefore call for evaluators and researchers to publish price and resource data on evaluations more widely, so that the evaluation community can examine capability progress in light of real-world economic constraints.