What really drives the cost of AI agents: the price per token, or how many tokens the agent burns through?
How do token-count variations affect cost estimates as AI workflows become more agentic?
This explores why token counts, more than price per token, end up driving what agentic AI actually costs, and how the right way to estimate that cost changes as agents run longer, cache more, and coordinate with each other.
This explores why token counts, more than price per token, end up driving what agentic AI costs, and how cost estimation has to change as workflows get longer and more autonomous. The short version from the corpus: in agentic systems, token use isn't a fixed overhead you multiply by a price. It swings widely depending on how the agent's surrounding software (its harness) is built, it is tightly linked to how well the agent performs, and past a certain point 'cost per token' stops being the useful unit.
Start with how large the swings are. Two separate lines of work show that the same task can use roughly half the tokens without retraining the model. Microsoft's ACON method only rewrites the instructions that tell an agent how to compress its own context, and it cut peak token use by 26–54% while also raising task success Can prompt optimization cut agent token usage without retraining?. Automated harness optimization across many environments found four mechanisms (how actions run, how context gets compacted, how tool outputs are handled, and handing reading off to helpers) that reduced token traffic by 44.7–49% at comparable performance Can agent harnesses be automatically optimized across many environments?. So a cost estimate built from one agent setup can be off by about 2× for another setup running the same model. The harness matters as much as the price sheet.
The complication is that tokens are also what buy performance. Anthropic's internal evaluations found that token spending alone explains about 80% of the performance differences between multi-agent research systems Does token spending drive multi-agent research performance?, How does test-time scaling work at the agent level?. Deep research agents show a similar pattern: answer quality rises with the number of search steps on a curve that looks like the one for reasoning tokens, then flattens out Does search budget scale like reasoning tokens for answer quality?, How does test-time scaling work for individual research agents?. Cutting tokens can therefore cut quality. Another finding helps here: what predicts performance isn't the raw token count but 'effective feedback,' meaning the useful, non-repeated information the agent actually acts on. That measure explains performance far better (R²≈0.94) than raw tokens or tool calls (R²≈0.33–0.42) Does raw token spending actually predict agent performance?. Taken together with the compression results, this suggests that much of an agent's token bill is waste you can trim, and the rest is spending that actually buys results.
This is why the unit of cost estimation is shifting. A 115-day case study of a persistent agent found that 82.9% of its tokens were cache reads, meaning reused context billed at much lower rates. The authors argue that the meaningful denominator becomes the finished artifact, not the token Do persistent agents really cost less per token?. Work on compact models makes the same point from the model side: cost and delay pile up across a whole multi-step episode, so a 35B model trained to follow through on long tasks can sit at the cheap end of the cost-performance trade-off and still compete with much larger models Does model efficiency matter more than peak capability for real work?. There's also a practical rule for anyone reading benchmarks: report token counts, not only dollar figures. Prices keep falling, so a frozen dollar cost goes stale while a token count can be re-priced at any time Should benchmarks report token counts instead of frozen costs?.
One thing you might not expect: as agents start managing money themselves, their own handling of budgets becomes part of the cost picture. In one study, 92% of humans went ahead with a $10.01 purchase on a $10 budget, while AI models uniformly refused. The gap closed only when the models were shown the human reasoning, not just the human choices Do AI models follow budgets more rigidly than humans do?. And at the organizational level, Deloitte argues that adding agents onto old workflows tends to fail Does agentic AI succeed through process redesign or workflow layering?. Read alongside the token findings, that suggests the biggest cost lever may be redesigning the workflow so agents waste fewer tokens, not finding a cheaper model. The corpus is thinner on concrete forecasting methods. It tells you which variables matter (harness design, caching, effective feedback, the full episode) but doesn't offer a ready-made formula.
Sources 12 notes
Microsoft's ACON method reduces agent token consumption and improves task success by iteratively refining compression guidelines through failure analysis, not weight updates. This approach works across proprietary LLM agents and costs under $2 per benchmark.
Cross-environment harness optimization yielded four mechanisms (action execution, context compaction, observation handling, delegated reading) that reduced token traffic by 44.7–49.0% while maintaining comparable performance on a 51-task benchmark, suggesting harness-level gains are orthogonal to model improvements.
Anthropic's internal evals show token spending alone accounts for 80% of performance variance in multi-agent research systems. Model capability upgrades deliver larger gains than doubling token budget, suggesting efficiency matters as much as quantity.
Research shows 80% of multi-agent performance variance comes from token budget, not coordination intelligence. LatentMAS and shared-KV-cache approaches offer ways to decouple performance gains from token costs.
Agentic deep research shows monotonic-to-diminishing-returns curves for search iterations, matching reasoning token scaling. This creates a new inference-compute axis: models can trade off reasoning budget against search budget to optimize answer quality.
Show all 12 sources
Research shows that deep research agents exhibit test-time scaling laws where search steps scale similarly to reasoning tokens, and live search outperforms memorized retrieval on knowledge-intensive tasks. Data efficiency is extreme—78 curated demonstrations outperform 10K samples for agency.
Effective Feedback Compute—crediting only informative, valid, non-redundant feedback retained for decisions—predicts performance (R²≈0.94) far better than raw tokens or tool calls (R²≈0.33–0.42). The scaling lever is feedback quality, not quantity of interaction.
A 115-day case study found 82.9% of tokens were cache reads. When context persists and reuses, the meaningful cost denominator becomes completed artifacts, not individual tokens.
Occamy-1.0, a 35B-parameter model further trained on execution-grounded data and long-horizon trajectories, achieves competitive performance with much larger models while sitting at the low-cost knee of the Pareto frontier, suggesting that training for coordination and follow-through substitutes for raw scale in multi-step work.
Inference prices fall over time, making frozen dollar-cost figures misleading for procurement decisions. Reporting input/output token counts alongside costs lets downstream developers recalculate expenses as prices change, keeping benchmarks useful.
When faced with a $10.01 purchase on a $10 budget, 92% of humans proceeded while AI models uniformly refused. Providing models with human reasoning—not just choices—closed the gap and restored pragmatic judgment.
Deloitte argues many agentic AI implementations fail because organizations layer agents onto old workflows rather than redesigning operations. Success requires agent-compatible data architectures, robust orchestration, and new management models; HPE and Toyota illustrate this approach.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- How we built our multi-agent research system
- Holistic Agent Leaderboard: The Missing Infrastructure for AI Agent Evaluation
- Single-Agent LLMs Outperform Multi-Agent Systems on Multi-Hop Reasoning Under Equal Thinking Token Budgets
- The Price of Progress: Price Performance and the Future of AI
- RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human experts
- From Web Search towards Agentic Deep Research: Incentivizing Search with Reasoning Agents
- SoL-Pi: Recursively Scaling Auto-Research Loops for Efficient Agent Harness
- Worse Together: How Performance Breaks Down in Multi-User Multi-Agent Teams