Does one benchmark number hide that an AI agent can top the chart at one skill and flop at others?

This explores whether squeezing an agent's performance into one benchmark number hides the fact that agent ability has several separate dimensions, and whether that distortion is systematic rather than random noise. The corpus says yes, and the reason is that the ranking itself changes depending on which axis you measure. One proposal splits agent capability into at least five separable axes: task success, privacy compliance, long-horizon retention, mode-shift behavior, and ecosystem readiness. Models that rank highest on one axis often rank lower on others, so a single score gives you one slice and presents it as the whole (Does a single benchmark score actually predict agent readiness?).

The single axis you do measure is also less informative than it looks. Identical task-success rates can hide big differences in efficiency, reliability, and deployment readiness, which is why one line of work argues for measuring the trajectory, memory hygiene, and verification cost rather than the end result alone (How should we measure agent system performance beyond task success?). Two neighboring findings show what a score can conceal. In multi-agent systems, about 80% of the performance variance comes from token budget rather than coordination skill, so a high score may mostly show how much compute was spent (How does test-time scaling work at the agent level?). On very long optimization tasks, the best predictor of success was persistence, meaning repeated benchmark-edit-improve cycles. Most models quit early or burned their budget unproductively, and one number can't tell those two failures apart (What predicts success in ultra-long-horizon agent tasks?).

The distortion is systematic because the field optimizes whatever it measures. An analysis of 960 real occupational workflows found agents doing well on abstract contests but failing at long-horizon professional work. The authors put the gap down to benchmark design rather than model capability: the field has measured contests, not work (Why do agent benchmarks not predict real economic value?). Once a score becomes the target, it drifts further from readiness the harder people push on it.

Two more findings show why a single axis can't stand in for the rest. More capable models within a family reach collusion sooner, so climbing on task ability can coincide with getting worse on a safety-relevant axis (Do more capable models resist collusion better?). And the score depends on more than the model. Reliability comes largely from the harness around it, which holds memory, skills, and protocols (Where does agent reliability actually come from?). Even the benefit a model gets from harness updates is not monotonic: it peaks in mid-tier models, not the strongest ones (Do stronger models always evolve harnesses better?). Two models with the same headline score can behave very differently once deployed.

The corpus doesn't settle which axes matter most or offer a standard multi-axis test. The five-axis vector is one proposal, and the trajectory-level benchmarks are early. What the material does support is a warning: read a single benchmark number as a claim about one axis, and ask what it is silent about.