Should you judge an AI agent by what it costs to finish a whole job, not per-token price or benchmarks?

This explores whether the total cost of getting a whole task finished, rather than per-token price or a benchmark score, should be the main yardstick for choosing between AI agents in production. The corpus says episode cost is the right unit of comparison, but it doesn't say cost decides the outcome on its own. It also has no direct data on how teams buy or choose agents, so what follows is inferred from the evidence on efficiency and evaluation.

The case for the episode as the unit is strong. Cost and latency pile up over every step of a multi-step job, so a model that looks cheap per call can be expensive per finished task. In one study, a 35B model trained on execution-grounded, long-horizon data sat at the low-cost knee of the cost-performance curve. It kept up with much larger models, which suggests follow-through can substitute for raw scale (Does model efficiency matter more than peak capability for real work?). The unit can also move past the episode. In a 115-day case study of a persistent agent, 82.9% of tokens were cache reads, so cost per token said little and cost per completed artifact was the number that meant something (Do persistent agents really cost less per token?).

Cost still can't be the deciding factor alone, because a low episode cost can mean the agent did less. One analysis attributes about 80% of multi-agent performance variance to token budget rather than coordination cleverness (How does test-time scaling work at the agent level?). If so, a cheaper agent may just be spending less and getting less, and comparisons only mean something at matched spend. Long optimization tasks show the flip side. The best predictor of success was persistence, meaning repeated benchmark-edit cycles within the time budget, and many models quit early or burned budget without progress (What predicts success in ultra-long-horizon agent tasks?). Quitting early makes an agent cheap and useless, so cost only counts once the work is actually completed. Results also swing a lot between runs of the same agent (Do frontier AI agents actually conduct novel research or just optimize?), so one episode's cost is a sample from a distribution.

Cost also isn't purely a property of the model. Automatically optimized harnesses found four mechanisms (action execution, context compaction, observation handling, delegated reading) that cut token traffic by 44.7–49.0% at comparable performance, and the gains were independent of model improvements (Can agent harnesses be automatically optimized across many environments?). This fits the broader finding that reliability comes from memory, skills and protocols built around the model, not from the model alone (Where does agent reliability actually come from?). So the thing being compared is really a model-plus-harness bundle, and two agents on the same model can have very different episode costs.

The last caveat is what the episodes are. Agents that win benchmark contests often fail long professional workflows, and the benchmark-to-value gap comes from what got measured, not from model capability (Why do agent benchmarks not predict real economic value?). Cost per benchmark episode inherits that flaw. The corpus's answer is that cost per completed, useful outcome becomes the decisive comparison only after you have pinned down what counts as done, held effort constant, and measured on work that resembles your own.