INQUIRING LINE

Chasing accuracy alone makes AI agents needlessly complex and pricey — optimizing for cost too can cut spending without hurting results.

How much cheaper do agents become when jointly optimizing cost and accuracy?

This explores how much money you can save on AI agents by tuning for cost and accuracy together, instead of chasing accuracy alone, and what the corpus says about where those savings come from.


This explores how much cheaper agents get when cost counts alongside accuracy instead of accuracy alone. The corpus has no single headline number like "joint optimization saves X%." It does show that the room for savings is large, and that most of it sits outside the model itself. The starting point is that accuracy-only leaderboards reward complexity. If cost doesn't count, the top agents pile on retries, extra calls and elaborate logic that add little. Plotting cost against accuracy and picking from the best trade-offs often lowers total cost without losing accuracy. The savings grow when an agent runs the same task many times Why do accuracy-only benchmarks hide the true cost of agents?.

The clearest number comes from a cost-controlled evaluation across 9 benchmarks. The most expensive model was on the best-trade-off curve in only 1 of them. The scaffold, meaning the prompts, tools and control logic wrapped around the model, changed cost by up to 9x for small accuracy gains How much does agent design choice actually cost?. This may be the most surprising finding: picking a harness that fits the model can matter as much as picking the model. Automated harness tuning shows the same pattern. When improvement loops ran across many environments, they found four mechanisms that cut token traffic by about 45-49% at similar performance: smarter action execution, context compaction, better handling of tool outputs, and handing reading off to a helper Can agent harnesses be automatically optimized across many environments?. A well-built harness can also raise accuracy on a fixed model without retraining Can execution harnesses lift model performance without retuning weights?.

Why is the gap so wide? Agents loop: they plan, call tools, read results and plan again. A small amount of waste in each step compounds across the whole run, so making the model cheaper per token barely matters next to cutting unneeded steps Why does agent efficiency differ from model size reduction?. One framework splits agent efficiency into three separate levers: memory (tokens), tool use (latency) and planning (steps). Improving one doesn't automatically improve the others Does agent efficiency really break down into three distinct components?. On the model side, a 35B model trained on long, multi-step work trajectories sits at the low-cost point on the trade-off curve while competing with much larger models Does model efficiency matter more than peak capability for real work?.

The less obvious lesson is that "cheaper" depends on what you count. In a 115-day study of a long-running agent, 82.9% of tokens were cached re-reads of earlier context. The study argues the useful unit of cost is the finished piece of work, not the token Do persistent agents really cost less per token?. That changes the question: an agent that looks expensive per token can be cheap per result. So the honest answer is up to roughly 9x from scaffold choice and about half the tokens from harness tuning, and how much that means in practice depends on whether you measure tokens, runs or results.


Sources 8 notes

Why do accuracy-only benchmarks hide the true cost of agents?

Agent benchmarks that score only accuracy obscure cost trade-offs, driving SOTA agents toward unnecessary complexity. Joint optimization of cost and accuracy across a Pareto frontier can reduce total cost while maintaining accuracy, especially when amortized over repeated runs.

How much does agent design choice actually cost?

Across 9 benchmarks, the costliest model occupied the Pareto frontier in only 1 case, while scaffold choice (prompts, tools, logic) can swing costs 9x for minimal accuracy gains. Model-scaffold matching matters as much as model selection alone.

Can agent harnesses be automatically optimized across many environments?

Cross-environment harness optimization yielded four mechanisms (action execution, context compaction, observation handling, delegated reading) that reduced token traffic by 44.7–49.0% while maintaining comparable performance on a 51-task benchmark, suggesting harness-level gains are orthogonal to model improvements.

Can execution harnesses lift model performance without retuning weights?

StateM improves Terminal-Bench 2.1 accuracy across multiple models by optimizing execution systems around frozen weights. The same runbook transfers to newer models without modification, achieving 95.3% on GPT-5.6 and lifting DeepSeek-V4 Flash by 5.4 points.

Why does agent efficiency differ from model size reduction?

Agentic systems consume resources exponentially through recursive loops, making per-token model efficiency marginal. True efficiency requires system-level trade-offs between task success and total cost across planning, memory, and tool use.

Show all 8 sources
Does agent efficiency really break down into three distinct components?

Research identifies memory compression, tool learning efficiency, and planning optimization as three structurally independent components, each with distinct cost profiles (tokens, latency, and steps). Improving one axis does not automatically improve the others, requiring holistic design.

Does model efficiency matter more than peak capability for real work?

Occamy-1.0, a 35B-parameter model further trained on execution-grounded data and long-horizon trajectories, achieves competitive performance with much larger models while sitting at the low-cost knee of the Pareto frontier, suggesting that training for coordination and follow-through substitutes for raw scale in multi-step work.

Do persistent agents really cost less per token?

A 115-day case study found 82.9% of tokens were cache reads. When context persists and reuses, the meaningful cost denominator becomes completed artifacts, not individual tokens.

Papers this line draws on 8

The research behind the notes this line reads — ranked by how closely each paper relates.