An AI that tops one benchmark often ranks near the bottom on another — a single score hides all of that.
How should single-axis benchmarks account for separable capability dimensions?
This explores how a single benchmark number can be redesigned — or supplemented — when the thing being measured is really several independent capabilities that don't move together.
This explores how a single benchmark number can be redesigned when the thing it's scoring is actually several independent capabilities that don't rise and fall together. The corpus's sharpest answer is that they mostly can't, and shouldn't try: agent capability is better described as a vector across separable axes than as a point on a line. One analysis decomposes it into at least five — task success, privacy compliance, long-horizon retention, mode-shift behavior, and ecosystem readiness — and finds that models topping one axis routinely rank low on another, which makes any single score systematically misleading for real deployment Does a single benchmark score actually predict agent readiness?. The lesson isn't 'weight the axes better'; it's that collapsing separable dimensions into one number destroys exactly the information a deployer needs.
The reason single scores mislead becomes concrete when you watch capability over time instead of in a snapshot. Short-interaction benchmarks simply don't predict long-horizon delegated performance: models that look identical on single-turn tasks diverge sharply by relay 25 of a 50-round-trip workflow, revealing degradation curves the one-shot score can't see Do short benchmarks predict how models perform over long workflows?. Instruction-following shows the same hidden structure — performance decays in different shapes (linear, exponential, or a threshold cliff) depending on model type, so a single density point conceals which regime a model is in How does instruction density affect model performance?. A benchmark that reports one number at one operating point is averaging over the very dimension where models differ.
There's a deeper version of the problem: two models can post identical metrics while being internally different in ways the metric can't touch. A network can hold all the linearly decodable features a task needs yet have fractured internal organization, leaving it fragile to perturbation and distribution shift that standard evaluation never probes Can models be smart without organized internal structure?. So separability isn't only about outward capability axes — it's also about the gap between 'scores well' and 'is actually robust,' another dimension a single axis silently folds away.
What the corpus offers instead of a better scalar is a set of methods for surfacing the missing axes. Open-world evaluation of messy, long-horizon tasks through qualitative log analysis — with cost reported explicitly — corrects distortions that auto-gradable benchmarks both overstate and understate, and catches emerging capabilities earlier Do automated benchmarks hide what frontier AI systems can really do?. Frontier exams can genuinely discriminate where older benchmarks saturate, but even a perfect expert-exam score wouldn't tell you about autonomous research or open-world problem-solving — that's a different axis entirely Can frontier exams really measure cutting-edge AI capability?. And some capabilities named as essential — hypothesis generation, experimental design, iterative self-correction — go essentially unmeasured by current benchmarks, so the honest move is to name them as separate axes rather than assume the headline score covers them What capabilities do AI systems need for autonomous science?.
The thread you might not have expected: separability isn't only an evaluation problem, it's a design principle that keeps paying off elsewhere. When researchers physically separate a decomposer from a solver, they find decomposition skill transfers across domains while solving skill doesn't — the two are genuinely different capabilities, and forcing them through one model causes interference Does separating planning from execution improve reasoning accuracy?. The same geometry shows up in training, where isolating task-specific parameters prevents multi-task fine-tuning from interfering across tasks Can isolating task-specific parameters prevent multi-task fine-tuning interference?. So the answer to the question is really an argument by analogy: the field keeps discovering that capabilities are separable in reasoning and in weights — and a benchmark that reports them as one axis is fighting that structure instead of measuring it.
Sources 9 notes
Capability decomposes into task success, privacy compliance, long-horizon retention, mode-shift behavior, and ecosystem readiness. Models ranked highest on one axis often rank lower on others, making single-score evaluations systematically misleading for real deployment.
DELEGATE-52 evaluated models across 50-round-trip relays and found short-interaction performance does not predict sustained delegation accuracy. Models ranking similarly on single-turn tasks diverged dramatically by relay 25, revealing degradation curves invisible to standard benchmarks.
IFScale benchmark shows three degradation patterns: linear (small models), exponential (mid-range), and threshold decay (reasoning models maintain ~150 instructions then fail steeply). Even best models reach only 68% accuracy at maximum density.
Models trained with SGD can contain all the linearly decodable features needed for a task while maintaining fundamentally broken internal organization. This makes them vulnerable to perturbation and distribution shift invisible to standard evaluation metrics.
Automated benchmarks both overstate and understate capability by privileging precisely-specified, auto-gradable tasks. Open-world evaluations of long-horizon messy tasks through qualitative log analysis—with cost explicitly reported—correct these distortions and catch emerging capabilities earlier.
Show all 9 sources
Humanity's Last Exam uses 3,000 expert-designed questions to expose capability gaps where MMLU saturates, showing real discrimination—but expert exam performance wouldn't indicate autonomous research or open-world problem-solving that matters for deployment.
The Virtuous Machines framework identifies hypothesis generation, experimental design, data analysis, and iterative self-correction as essential for autonomous scientific research, none of which standard LLM benchmarks reliably evaluate. Self-correction poses the deepest challenge due to documented degradation in reasoning accuracy.
Modular architectures with separate decomposer and solver models outperform monolithic LLMs, with decomposition ability transferring across domains while solving ability does not. The separation prevents planning-execution interference and produces more generalizable skills.
Research shows that identifying core parameter regions per task, clustering overlapping tasks, and freezing core parameters while geometrically merging non-core parameters consistently outperforms standard multi-task fine-tuning. Temporal task scheduling alone proves insufficient without explicit structural parameter isolation.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Open-World Evaluations for Measuring Frontier AI Capabilities
- Agents' Last Exam
- AutoLab: Can Frontier Models Solve Long-Horizon Auto Research and Engineering Tasks?
- Scaling Reasoning, Losing Control: Evaluating Instruction Following in Large Reasoning Models
- Comprehension Without Competence: Architectural Limits of LLMs in Symbolic Computation and Reasoning
- LiveMCP-101: Stress Testing and Diagnosing MCP-enabled Agents on Challenging Queries
- LLMs Corrupt Your Documents When You Delegate
- A Survey on Post-training of Large Language Models