Should AI benchmarks score how good an answer is, or also how much it cost to get there?
How can benchmarks optimize for both accuracy and computational cost simultaneously?
This explores how AI evaluations can measure accuracy and the compute or money spent to get it together, rather than ranking systems on accuracy alone, and what the corpus says happens when cost is left out.
This explores how benchmarks could score AI systems on accuracy and on what that accuracy costs, instead of ranking on accuracy alone. In the corpus, the usual answer is to stop reporting a single number. Report a frontier instead: plot every system by cost and accuracy, and treat as best only the systems that no other system beats on both at once. Agent benchmarks show the problem clearly. When leaderboards reward only accuracy, top agents pile on retries, extra model calls and elaborate scaffolding that cost a lot and add little. Optimizing cost and accuracy together along that frontier can cut total spend without losing accuracy, and the savings grow when an agent is designed once and run many times Why do accuracy-only benchmarks hide the true cost of agents?.
The reason this matters more than it sounds: accuracy-only scores can quietly measure spending instead of intelligence. Frontier models' benchmark performance per dollar improves 5–10× a year, yet the absolute cost of running them rises 3–18× a year. On one hard science benchmark (GPQA-Diamond), roughly half of the measured progress tracks more inference spending rather than gains that hold at fixed compute Is benchmark progress outpacing the cost to achieve it?. A leaderboard that hides cost can't separate "this model got smarter" from "this model was allowed to think longer." A related critique says automated benchmarks also distort capability itself, because they favor neatly specified, auto-gradable tasks. The proposed fix is open-world evaluation of messy long-horizon tasks, with cost reported explicitly alongside the results Do automated benchmarks hide what frontier AI systems can really do?.
Once cost is on the scoreboard, a whole family of techniques becomes visible, because they win on the cost axis rather than the accuracy axis. Giving easy prompts less compute and hard prompts more beats a uniform budget, and can even beat a larger model given the same total compute Can we allocate inference compute based on prompt difficulty? How should we spend compute at inference time?. Watching confidence step by step lets a system abandon failing reasoning attempts early. It matches the accuracy of large majority-vote ensembles while generating far fewer attempts Does step-level confidence outperform global averaging for trace filtering?. At the architecture level, scaling laws that include design choices such as hidden size and the ratio of MLP to attention layers found models that were both slightly more accurate and 42% faster to run Can architecture choices improve inference efficiency without sacrificing accuracy?. Pairing cheap lookup memory with Mixture-of-Experts routing beats pure Mixture-of-Experts at the same parameter count and FLOPs Can lookup memory and computation work together better than either alone?. An accuracy-only leaderboard would undervalue every one of these.
There's a limit, though: cost-aware scoring doesn't mean any model can buy its way up the curve. Non-reasoning models can't match reasoning models even with unlimited inference budget, because training is what makes extra tokens useful Can non-reasoning models catch up with more compute?. A good cost–accuracy benchmark therefore shows two things: where spending more helps, and where a system has hit a wall that more spending can't fix.
The less obvious point is that whoever designs a benchmark is choosing what gets built. Systems that improve themselves, like the Darwin Gödel Machine, use benchmark scores as their only fitness signal Can AI systems improve themselves through trial and error?, and harness-engineering work improves frozen models by tuning the scaffolding around them Can execution harnesses lift model performance without retuning weights?. If those loops optimize for accuracy alone, they will drift toward expensive complexity. Adding cost to the objective changes what they discover. The corpus doesn't settle on one standard way to combine the two numbers, but it agrees that reporting both, as a frontier, is the minimum.
Sources 11 notes
Agent benchmarks that score only accuracy obscure cost trade-offs, driving SOTA agents toward unnecessary complexity. Joint optimization of cost and accuracy across a Pareto frontier can reduce total cost while maintaining accuracy, especially when amortized over repeated runs.
Regression analysis of frontier model pricing shows benchmark performance per dollar improves 5–10× annually, yet the absolute cost to run these models rises 3–18× annually. Roughly half of measured progress on GPQA-Diamond tracks increased inference spending rather than capability gains independent of compute.
Automated benchmarks both overstate and understate capability by privileging precisely-specified, auto-gradable tasks. Open-world evaluations of long-horizon messy tasks through qualitative log analysis—with cost explicitly reported—correct these distortions and catch emerging capabilities earlier.
Research shows inference effectiveness varies dramatically by prompt difficulty. Reallocating the same total compute adaptively—giving easy prompts less and hard ones more—substantially outperforms larger models under uniform budgets.
Research shows that uniform inference budgets waste compute; allocation should vary by prompt. Test-time compute can substitute for training-time scaling on hard problems, but cannot overcome fundamental limitations set by the training regime.
Show all 11 sources
Local step-level confidence catches reasoning breakdowns that global averaging masks and enables early stopping before traces complete. This approach achieves comparable accuracy gains to naive majority voting with far fewer generated traces, proving trace quality matters more than quantity.
Augmenting scaling laws with hidden size, MLP-to-attention ratio, and GQA configuration enables architecture optimization for inference. Optimized models achieved up to 2.1% higher accuracy and 42% greater throughput than LLaMA-3.2 under identical training budgets.
Engram combines O(1) N-gram lookup with Mixture-of-Experts routing, revealing a U-shaped scaling law where balanced allocation to both mechanisms outperforms either alone. Gains appear largest in reasoning and code rather than pure retrieval.
Reasoning models persistently outperform non-reasoning models regardless of inference budget because training instills a reasoning protocol that makes additional tokens productive. The gap is fundamentally about deployment mechanisms and training structure, not raw capability.
DGM replaces formal proofs with empirical benchmarking and maintains an evolutionary archive of agent variants, achieving 2.5× improvement on SWE-bench and 2.2× on Polyglot by discovering capabilities like better code editing and context management.
StateM improves Terminal-Bench 2.1 accuracy across multiple models by optimizing execution systems around frozen weights. The same runbook transfers to newer models without modification, achieving 95.3% on GPT-5.6 and lifting DeepSeek-V4 Flash by 5.4 points.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- When More Thinking Hurts: Overthinking in LLM Test-Time Compute Scaling
- Single-Agent LLMs Outperform Multi-Agent Systems on Multi-Hop Reasoning Under Equal Thinking Token Budgets
- The Price of Progress: Price Performance and the Future of AI
- Think Twice: Enhancing LLM Reasoning by Scaling Multi-round Test-time Thinking
- PostTrainBench: Can LLM Agents Automate LLM Post-Training?
- Holistic Agent Leaderboard: The Missing Infrastructure for AI Agent Evaluation
- Reasoning Models Can Be Effective Without Thinking
- Does Thinking More always Help? Understanding Test-Time Scaling in Reasoning Models