If an AI scores well on one big test, could it still be failing at things that test never checks?
Do multi-axis benchmarks reveal failures that single-axis benchmarks systematically hide?
This explores whether scoring AI systems on several separate dimensions exposes weaknesses that a single headline number consistently covers up, and what those hidden failures look like.
This explores whether scoring AI systems on several separate dimensions exposes weaknesses that one headline number consistently covers up. The corpus says yes, and it shows the hiding happens in a few distinct ways. It also warns that adding axes doesn't make evaluation easy.
The most direct evidence is Does a single benchmark score actually predict agent readiness?. It splits agent readiness into at least five separable axes: task success, privacy compliance, long-horizon retention, mode-shift behavior, and ecosystem readiness. Models that top one axis often rank lower on another. So a single score can give you the wrong ordering when you're deciding what to deploy, not just a blurry one. This is the only note in the set that compares axes head to head. The rest support the idea from other angles.
One way a single number hides failure is by mixing different causes into one value. On the exploitation benchmark in What causes failures in exploitation benchmarks?, a zero can mean the model refused for safety reasons, a tool broke, or the vulnerability was impossible. The note treats low scores as a lower bound on capability, not a measurement of it. The natural fix is to report those causes separately.
A second way is when failures look like success. Does model capability change how documents degrade? finds that weaker models visibly delete content, while frontier models silently corrupt it and keep the document looking intact. A check that only asks whether the output still looks fine would favor the frontier models exactly where their failures are hardest to catch. A third way is when a failure only shows up over time. In Do models fail worse when their own errors fill the context?, errors piling up in context degrade performance non-linearly, and bigger models don't fix it. A short single-step test never runs long enough to see this, which is why long-horizon retention counts as its own axis.
The same pattern appears at the smallest scale. Does step-level confidence outperform global averaging for trace filtering? shows that averaging confidence over a whole reasoning trace masks the local step where the reasoning broke, while step-level scoring catches it. Averaging across steps or across axes gives you one number, and it can hide the breakdown you most needed to see.
The catch is in Do interactive evaluations actually solve the benchmark comparison problem?. Richer, interactive evaluations don't retire the old problems. Comparability, reproducibility, and turning evidence into a judgment come back in a higher-dimensional form. More axes give you more places to spot a failure, but also more ways for two labs to disagree about what a score means. Multi-axis benchmarks do expose what single scores hide, but only if the field also agrees on shared design protocols for reading them.
Sources 6 notes
Capability decomposes into task success, privacy compliance, long-horizon retention, mode-shift behavior, and ecosystem readiness. Models ranked highest on one axis often rank lower on others, making single-score evaluations systematically misleading for real deployment.
ExploitGym's zero scores mix together safety refusals, tool errors, and impossible vulnerabilities—making low scores a lower bound rather than a true measure of capability, particularly problematic when assessing agent danger.
DELEGATE-52 shows weaker LLMs degrade documents through visible deletion, while frontier models degrade through subtle corruption that preserves surface integrity. This shift makes frontier failures harder to detect and potentially more dangerous at workflow scale.
Error accumulation in context causes non-linear performance degradation in long-horizon tasks. Model scaling does not fix this; only test-time compute through thinking models reduces the effect by preventing error-contaminated context from biasing reasoning.
Local step-level confidence catches reasoning breakdowns that global averaging masks and enables early stopping before traces complete. This approach achieves comparable accuracy gains to naive majority voting with far fewer generated traces, proving trace quality matters more than quantity.
Show all 6 sources
Interactive evaluation relocates core problems—comparability, reproducibility, evidence-to-judgment mapping—into higher-dimensional space rather than solving them. The field needs design protocols and shared standards, not format adoption, to make trajectory scoring interpretable.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- The Illusion of Diminishing Returns: Measuring Long Horizon Execution in LLMs
- LLMs Corrupt Your Documents When You Delegate
- Large Language Model Reasoning Failures
- LiveMCP-101: Stress Testing and Diagnosing MCP-enabled Agents on Challenging Queries
- Open-World Evaluations for Measuring Frontier AI Capabilities
- Deep Think with Confidence
- Local Coherence or Global Validity? Investigating RLVR Traces in Math Domains
- Interactive Evaluation Requires a Design Science