Running AI agents isn't like asking a chatbot a question once — it's a meter that keeps running the whole time the agent works.
What inference costs does running AI agents actually incur daily?
This explores what it actually costs to keep AI agents running day to day, and what drives that cost. The corpus has no dollar-per-day figures, but it says a lot about where the money goes and why the usual way of measuring it misleads.
This explores the real, ongoing cost of running AI agents and what drives it. One limit first: the collection doesn't give daily dollar figures for agent deployments. What it does show is where agent inference costs come from, why counting tokens can mislead, and which design choices shrink the bill. That is arguably more useful than a number that would be out of date within a quarter.
Start with the scale shift. A chatbot uses compute only when someone asks it something. An autonomous agent keeps consuming inference for as long as it runs. That's why one analysis of the AI compute stack argues inference, not training, will become the dominant demand as agents and robots spread Will inference compute soon exceed training compute demand?. Costs also pile up across a whole task rather than per answer. Every planning step, tool call and re-read of context adds to the bill, so a model that's slightly cheaper per call but finishes multi-step work in fewer steps can win overall. A 35B-parameter model trained specifically on execution and follow-through reached the low-cost end of the cost-performance curve this way Does model efficiency matter more than peak capability for real work?.
The most surprising finding is what the tokens actually are. A 115-day case study of a long-running agent found that 82.9% of its tokens were cache reads: the agent was re-reading context it already had, not producing new work. Cached tokens are much cheaper, so cost per token stops being a useful measure. The study suggests counting cost per finished artifact instead Do persistent agents really cost less per token?. Much of the cost also sits in the harness, the scaffolding around the model that manages context, tool outputs and file reading. Automated harness optimization across many environments found four mechanisms that cut token traffic by 44.7–49.0% with comparable performance, and those savings are independent of which model you use Can agent harnesses be automatically optimized across many environments?. Harness work can also raise accuracy on frozen models Can execution harnesses lift model performance without retuning weights?, so the same scaffolding can lift capability and cut cost.
Why don't more teams notice this? Benchmarks mostly don't measure cost. Agent leaderboards that score only accuracy push state-of-the-art agents toward needless complexity. Optimizing cost and accuracy together can cut total spend sharply, especially when a task runs over and over Why do accuracy-only benchmarks hide the true cost of agents?. Benchmarks also reward contest-style tasks over the long, messy workflows where real costs pile up Why do agent benchmarks not predict real economic value?.
The practical levers fit together. Route most routine subtasks to small models, which handle repetitive, well-defined work at 10–30× lower cost, and call a large model only when needed Can small language models handle most agent tasks?. Spend more compute on hard prompts and less on easy ones instead of a flat budget for everything Can we allocate inference compute based on prompt difficulty?. Keep the limits of cheap fixes in mind, though. A non-reasoning model can't close the gap with a reasoning model just by getting more inference budget, because the extra tokens only pay off if training taught the model how to use them Can non-reasoning models catch up with more compute?. The takeaway: an agent's daily bill depends less on token price than on how much context it re-reads, how well its harness is built, and whether cheap models handle the routine work.
Sources 10 notes
A seven-layer AI compute stack model reveals that inference demand grows along a new axis training never had: every autonomous agent is a continuous inference consumer. This makes inference, not training, the likely dominant compute demand as agents and robots scale.
Occamy-1.0, a 35B-parameter model further trained on execution-grounded data and long-horizon trajectories, achieves competitive performance with much larger models while sitting at the low-cost knee of the Pareto frontier, suggesting that training for coordination and follow-through substitutes for raw scale in multi-step work.
A 115-day case study found 82.9% of tokens were cache reads. When context persists and reuses, the meaningful cost denominator becomes completed artifacts, not individual tokens.
Cross-environment harness optimization yielded four mechanisms (action execution, context compaction, observation handling, delegated reading) that reduced token traffic by 44.7–49.0% while maintaining comparable performance on a 51-task benchmark, suggesting harness-level gains are orthogonal to model improvements.
StateM improves Terminal-Bench 2.1 accuracy across multiple models by optimizing execution systems around frozen weights. The same runbook transfers to newer models without modification, achieving 95.3% on GPT-5.6 and lifting DeepSeek-V4 Flash by 5.4 points.
Show all 10 sources
Agent benchmarks that score only accuracy obscure cost trade-offs, driving SOTA agents toward unnecessary complexity. Joint optimization of cost and accuracy across a Pareto frontier can reduce total cost while maintaining accuracy, especially when amortized over repeated runs.
ALE's analysis of 960 real occupational workflows shows agents excel at abstract contests but fail long-horizon professional tasks. The gap is not model capability but benchmark design—the field optimizes what it measures, and it has measured contests rather than work.
SLMs handle the repetitive, well-defined language tasks that constitute most agent work at 10–30× lower cost than LLMs, making heterogeneous architectures (SLMs by default, LLMs selective) the economically rational design pattern.
Research shows inference effectiveness varies dramatically by prompt difficulty. Reallocating the same total compute adaptively—giving easy prompts less and hard ones more—substantially outperforms larger models under uniform budgets.
Reasoning models persistently outperform non-reasoning models regardless of inference budget because training instills a reasoning protocol that makes additional tokens productive. The gap is fundamentally about deployment mechanisms and training structure, not raw capability.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Holistic Agent Leaderboard: The Missing Infrastructure for AI Agent Evaluation
- AI Agents That Matter
- Survey on Evaluation of LLM-based Agents
- PostTrainBench: Can LLM Agents Automate LLM Post-Training?
- Harness Updating Is Not Harness Benefit: Disentangling Evolution Capabilities in Self-Evolving LLM Agents
- Reasoning Models Can Be Effective Without Thinking
- Single-Agent LLMs Outperform Multi-Agent Systems on Multi-Hop Reasoning Under Equal Thinking Token Budgets
- Is a compute crunch coming?