Should an AI benchmark score come with a price tag showing how much compute it took to get there?
Should benchmark scores be published alongside their inference cost?
This explores whether a benchmark score means much without knowing how much compute and money it took to get it, and what form that cost reporting should take.
This explores whether a benchmark score means much without knowing how much compute and money it took to get it. The corpus says yes, firmly. It also says the obvious way to do it, printing a dollar figure next to the score, isn't enough.
The strongest case comes from looking at where benchmark progress actually comes from. One analysis of frontier model pricing found that performance per dollar improves 5 to 10 times a year, while the absolute cost of running top models rises 3 to 18 times a year. About half of the measured progress on a hard science benchmark (GPQA-Diamond) tracks increased inference spending rather than gains that hold independent of compute Is benchmark progress outpacing the cost to achieve it?. If you only see the score, you can't tell whether a model got smarter or just thought longer. Test-time scaling research makes that ambiguity hard to ignore. The same total compute can be spread evenly across prompts or concentrated on the hard ones, and adaptive allocation can beat a larger model run on a uniform budget Can we allocate inference compute based on prompt difficulty? How should we spend compute at inference time?. So a score is partly a statement about how the model was spent, not only about what the model is.
The less obvious point: dollar costs go stale. Inference prices keep falling, so a cost frozen at publication time quickly misleads anyone using it for purchasing decisions. The fix is to report input and output token counts alongside cost, so readers can recalculate at today's prices Should benchmarks report token counts instead of frozen costs?. Tokens are the durable unit. Dollars are a snapshot.
Cost is one piece of a wider argument about provenance. A score stripped of its settings (which prompts, which harness, how many attempts, how much compute) can't support a fair comparison. That's the premise behind efforts to keep source details attached to every reported number Can benchmark scores be trusted without knowing their origin?. Contamination is a sharp reminder of what goes wrong without that context: one math model could reconstruct over half of a popular test set from partial prompts but scored zero on a fresh one Does RLVR success on math benchmarks reflect genuine reasoning improvement?. Moving to richer agent-style evaluations doesn't escape this either. The same comparability problems come back at the level of whole multi-step runs Do interactive evaluations actually solve the benchmark comparison problem?.
There's a limit worth knowing about, though. Cost reporting tells you what a score cost, but it doesn't tell you how far spending more could take you. Models without reasoning training can't catch up to reasoning models even with unlimited inference budget, because the training is what makes extra tokens useful Can non-reasoning models catch up with more compute?. Efficiency can also come from places a token count won't show. Architecture choices alone delivered 42% higher throughput at equal or better accuracy Can architecture choices improve inference efficiency without sacrificing accuracy?. Smarter filtering of reasoning traces matched majority-vote accuracy with far fewer generated attempts Does step-level confidence outperform global averaging for trace filtering?. The takeaway: publish token counts, not just dollars, and treat score-plus-cost as a curve you can compare across models, not a single point.
Sources 10 notes
Regression analysis of frontier model pricing shows benchmark performance per dollar improves 5–10× annually, yet the absolute cost to run these models rises 3–18× annually. Roughly half of measured progress on GPQA-Diamond tracks increased inference spending rather than capability gains independent of compute.
Research shows inference effectiveness varies dramatically by prompt difficulty. Reallocating the same total compute adaptively—giving easy prompts less and hard ones more—substantially outperforms larger models under uniform budgets.
Research shows that uniform inference budgets waste compute; allocation should vary by prompt. Test-time compute can substitute for training-time scaling on hard problems, but cannot overcome fundamental limitations set by the training regime.
Inference prices fall over time, making frozen dollar-cost figures misleading for procurement decisions. Reporting input/output token counts alongside costs lets downstream developers recalculate expenses as prices change, keeping benchmarks useful.
Benchmark Radar catalogs AI evaluations while keeping source identities and citations attached, enabling readers to trace scores back to their original settings. The system demonstrates that scores without provenance cannot reliably support model comparisons.
Show all 10 sources
Qwen2.5-Math-7B reconstructs 54.6% of MATH-500 from partial prompts but scores 0.0% on post-release LiveMathBench, revealing dataset contamination. On clean benchmarks, only correct rewards improve performance; random and inverse rewards fail or degrade reasoning ability.
Interactive evaluation relocates core problems—comparability, reproducibility, evidence-to-judgment mapping—into higher-dimensional space rather than solving them. The field needs design protocols and shared standards, not format adoption, to make trajectory scoring interpretable.
Reasoning models persistently outperform non-reasoning models regardless of inference budget because training instills a reasoning protocol that makes additional tokens productive. The gap is fundamentally about deployment mechanisms and training structure, not raw capability.
Augmenting scaling laws with hidden size, MLP-to-attention ratio, and GQA configuration enables architecture optimization for inference. Optimized models achieved up to 2.1% higher accuracy and 42% greater throughput than LLaMA-3.2 under identical training budgets.
Local step-level confidence catches reasoning breakdowns that global averaging masks and enables early stopping before traces complete. This approach achieves comparable accuracy gains to naive majority voting with far fewer generated traces, proving trace quality matters more than quantity.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- When More Thinking Hurts: Overthinking in LLM Test-Time Compute Scaling
- Think Twice: Enhancing LLM Reasoning by Scaling Multi-round Test-time Thinking
- The Price of Progress: Price Performance and the Future of AI
- Reasoning Models Can Be Effective Without Thinking
- Does Thinking More always Help? Understanding Test-Time Scaling in Reasoning Models
- Local Coherence or Global Validity? Investigating RLVR Traces in Math Domains
- SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-training
- Inference-Aware Prompt Optimization for Aligning Black-Box Large Language Models