Why can an AI agent top a leaderboard yet still be a bad bet to actually deploy?

This explores why one headline number, usually task success on a single benchmark, can't tell you whether an agent is ready for real use. The short answer from the corpus is that readiness is several separable things at once. One score flattens them, and the rankings can flip when you look at the parts. One proposal splits capability into at least five axes: task success, privacy compliance, long-horizon retention, behavior when the mode of work shifts, and ecosystem readiness. Models that top one axis often rank lower on others, so a single-score leaderboard can be systematically misleading. Does a single benchmark score actually predict agent readiness?

Even inside the success axis, the number hides how the agent got there. Identical success rates can mask huge differences in efficiency, reliability, and verification cost, which is why the argument is for benchmarks that score the whole trajectory, not just the endpoint. How should we measure agent system performance beyond task success? Memory shows the same pattern. Breaking it into storage, extraction, retrieval, and maintenance shows which stage actually failed, while an end-to-end score only tells you that something did. How should we actually evaluate agent memory systems? A pass/fail score is like a car that arrived late with no record of whether the engine, the route, or the driver was at fault.

A second problem is that a score often measures more than the model. Wrapping frozen weights in a better execution harness lifted several models on Terminal-Bench 2.1 with no retraining, and the same runbook carried over to newer models. Can execution harnesses lift model performance without retuning weights? That fits the view that agent reliability comes largely from externalizing memory, skills, and protocols into the harness, not from raw model scale. Where does agent reliability actually come from? Spend matters too. In multi-agent setups, about 80% of the performance variance traced to token budget, not coordination intelligence, so a high score may mostly show how much compute was spent. How does test-time scaling work at the agent level? A single-axis benchmark can't separate the model, the scaffolding, and the bill.

The last gap is between what benchmarks measure and what deployment requires. On 960 real occupational workflows, agents that excel at abstract contests struggled with long-horizon professional tasks. The problem was benchmark design: the field optimizes what it measures, and it has measured contests instead of work. Why do agent benchmarks not predict real economic value? GUI agents show a similar pattern. Ones tuned for simulated environments stumble in the wild, where real-device runtimes and co-designed data and training are needed. Why do GUI agents fail when leaving the lab? Beyond the lab, a historical analysis suggests that capable agents stall when five conditions are missing: value generation, personalization, trustworthiness, social acceptability, and standardization. No task-success benchmark can register those. Why do capable AI agents still fail in real deployments?

In practice, a benchmark score tells you how a model-plus-harness-plus-budget combination did on one kind of task, judged at the finish line. Deployment readiness also depends on how the agent behaves along the way, how it handles long and messy real work, and whether the surrounding ecosystem can absorb it.