One score can crown the 'best' AI agent, yet the winner may be the wrong pick for your actual job.
Can a single leaderboard score capture multi-dimensional differences in agent performance?
This explores whether ranking agents by one leaderboard number can represent the many ways agents differ (efficiency, reliability, privacy, persistence), and what the corpus suggests measuring instead.
This explores whether one leaderboard number can stand in for the many ways agents differ. The corpus says no, and gives several separate reasons. The most direct is that agent capability behaves like a vector, not a scalar. One note splits it into at least five axes: task success, privacy compliance, long-horizon retention, mode-shift behavior, and ecosystem readiness. The model that tops one axis often ranks lower on another, so a single-score ranking can point you at the wrong agent for a real deployment (Does a single benchmark score actually predict agent readiness?).
The same success rate can also hide very different agents. Two agents can finish the same share of tasks while differing enormously in efficiency, reliability, memory hygiene, and the cost of verifying their work (How should we measure agent system performance beyond task success?). This is why evaluation is shifting from final answers to whole interaction trajectories, scored on process quality, recoverability, coordination, and robustness (How should we evaluate agent behavior beyond final answers?). One trajectory-level finding is that on ultra-long optimization tasks, persistence predicted success better than initial quality. Most of the 17 frontier models quit early or burned their budget unproductively, and a final score would not show that (What predicts success in ultra-long-horizon agent tasks?).
The score also isn't purely about the model. Optimizing the execution harness around frozen weights lifted several models on Terminal-Bench 2.1, including a 5.4-point gain for DeepSeek-V4 Flash, and the same runbook transferred to newer models unchanged (Can execution harnesses lift model performance without retuning weights?). That fits a wider claim that agent reliability comes from externalizing memory, skills, and protocols into the harness (Where does agent reliability actually come from?). A leaderboard row therefore blends the model with its scaffolding. AgentCompass responds by splitting evaluation into separate benchmark, harness, and environment components. This makes failures like reward hacking visible, where a scalar score conceals them (How can we make reward-hacking visible in agent evaluation?).
The deepest problem is what the number rewards. An analysis of 960 real occupational workflows found agents that win abstract contests but fail long-horizon professional tasks. The authors trace this to benchmark design, not model weakness: the field optimizes what it measures, and it has measured contests rather than work (Why do agent benchmarks not predict real economic value?). A leaderboard turns into a target, and a single target gets gamed.
Systems that have to act on agent quality already avoid the scalar. Capability-vector routing matches tasks to agents by semantic profile, with policy and budget constraints attached (Can semantic capability vectors replace manual agent routing?). Team-level methods score each agent's contribution and drop the uninformative ones (Can multi-agent teams automatically remove their weakest members?). Both treat differences between agents as the signal, and a single team score would erase them. The corpus doesn't say how to weigh the axes when you must pick one agent. Its consistent advice is to report a profile, not a rank, and to publish trajectories alongside it.
Sources 10 notes
Capability decomposes into task success, privacy compliance, long-horizon retention, mode-shift behavior, and ecosystem readiness. Models ranked highest on one axis often rank lower on others, making single-score evaluations systematically misleading for real deployment.
Single task-success metrics obscure how agents achieve results across memory, context, and verification layers. Research shows identical success rates can mask enormous differences in efficiency, reliability, and deployment readiness—requiring harness-level benchmarks that measure trajectory, memory hygiene, and verification costs.
Evaluation of agentic systems shifts evidence from final responses to full interaction sequences, and scoring procedure from correctness alone to process quality, recoverability, coordination, and robustness. This pattern appears across multiple agent benchmarks as a coherent design move.
Across 17 frontier models on 36 expert-curated optimization tasks, repeated benchmark-edit-incorporate cycles within a wall-clock budget proved the dominant success predictor. Most models terminated early or burned budget unproductively; Claude Opus 4.6 stood out as persistent.
StateM improves Terminal-Bench 2.1 accuracy across multiple models by optimizing execution systems around frozen weights. The same runbook transfers to newer models without modification, achieving 95.3% on GPT-5.6 and lifting DeepSeek-V4 Flash by 5.4 points.
Show all 10 sources
Research shows reliable LLM agents externalize three cognitive burdens—memory (state persistence), skills (procedural components), and protocols (structured interaction)—into a harness layer rather than relying on model scale alone. The harness unifies these externalities and eliminates the need for the model to solve the same problems repeatedly.
AgentCompass separates benchmark, harness, and environment into independent components, enabling trajectory analysis that surfaces reward-hacking and other failure modes scalar scores conceal. This architectural change shifts evaluation from opaque final scores to inspectable agent behavior.
ALE's analysis of 960 real occupational workflows shows agents excel at abstract contests but fail long-horizon professional tasks. The gap is not model capability but benchmark design—the field optimizes what it measures, and it has measured contests rather than work.
Versioned Capability Vectors embedded in HNSW indices couple semantic matching with policy and budget constraints, making capability discovery a first-class operation that scales sub-linearly as agent heterogeneity increases.
DyLAN's three-step importance scoring mechanism (propagation, aggregation, selection) quantifies individual agent contributions and automatically removes uninformative agents during inference, optimizing team composition without task-specific tuning.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Survey on Evaluation of LLM-based Agents
- BenchShield: Formal Model-Backed Instrumentation for Reward Integrity in LLM-Agent Evaluation Infrastructure
- AgentCompass: A Unified Evaluation Infrastructure for Agent Capabilities
- AutoLab: Can Frontier Models Solve Long-Horizon Auto Research and Engineering Tasks?
- LLMs Corrupt Your Documents When You Delegate
- LiveMCP-101: Stress Testing and Diagnosing MCP-enabled Agents on Challenging Queries
- Towards a Science of Scaling Agent Systems
- From Model Scaling to System Scaling: Scaling the Harness in Agentic AI