Line of inquiry
Inquiring lines›How can we ensure AI systems remai…›How should autonomy and oversight…›this line of inquiry
How do standardized benchmarks differ from measures of real-world AI capability?
A broader line of inquiry — a family of 68 specific questions the research asks around this. Follow one into its inquiring-line page, or move sideways to a related line below.
Questions in this line of inquiry 68
Specific inquiring lines the field asks around this — ordered from the most general framing down to the most specific angle.
- Do single-axis benchmarks systematically misrepresent what AI systems can actually do?
- How do live human evaluations differ from ground-truth benchmarks?
- Why do AI benchmarks measure accuracy instead of reasoning quality?
- Could deliberate training toward benchmarks overstate real gains in AI capability?
- Why do open-world evaluations reveal capabilities that static benchmarks hide?
- How do existing evaluations measure AI capability in contained environments?
- How much do internal benchmarks differ from independent model evaluations?
- Why do AI benchmarks with verifiable rewards miss important capabilities?
- How do open-world evaluations correct distortions that automated benchmarks introduce?
- What gap exists between AI model capability in benchmarks and real client work?
- How can high benchmark performance mask broken reasoning in AI systems?
- Why do benchmark tasks differ from real occupational workflows in practice?
- Can automated benchmarks accurately capture progress on real-world long-horizon tasks?
- Can optimization metrics hide actual versus apparent progress in AI systems?
- Should evaluations shift toward open-world messy tasks instead of contests?
- What explains the gap between benchmark task horizons and real workplace autonomy?
- Do standard benchmarks miss how humans actually fail to use AI advice?
- Do multi-axis benchmarks reveal failures that single-axis benchmarks systematically hide?
- How do lab-scale benchmark tasks differ from real frontier AI research?
- How should domain-specific AI be evaluated differently from general benchmarks?
- Do bounded tasks let AI close performance gaps but open tasks widen them?
- How can benchmarks optimize for both accuracy and computational cost simultaneously?
- How do benchmark environments misrepresent deployment readiness?
- Does higher capability correlate with more benchmark contamination?
- How do surface correlations between narratives and answers mislead benchmark validity?
- Do automated benchmarks accurately measure real-world strategic reasoning ability?
- Do gains in optimization benchmark scores translate to gains in real research efficiency?
- Do narrow benchmark improvements translate directly to broader economic capability gains?
- How should human-AI evaluation differ from standalone model benchmarks?
- Can self-administered surveys establish trustworthy AI capability benchmarks?
- Can single performance scores hide important differences in how agents approach research tasks?
- Can capability indices reliably track real autonomous research ability?
- Why do benchmark scores not capture the true nature of AI systems?
- How do narrow benchmark optimizations differ from genuine architectural research discoveries?
- Can open-world evaluations become a scalable paradigm without becoming the next benchmark trap?
- Why do static benchmarks miss frontier capabilities that open-world tasks reveal?
- Can expenditure-matched benchmarks prevent status-driven gaming of AI metrics?
- Why do AI agents outpace human benchmarks yet fail user acceptance?
- Do success-only evaluations systematically overestimate real-world deployment readiness?
- How should evaluation frameworks account for the computational cost of frontier AI capability?
- How do single average metrics conceal rare but severe AI failures?
- Why do most self-improving systems fail when given tasks with no clear external benchmark?
- Does ARC performance on novel tasks predict real-world usefulness?
- What distinct domains of AI competence do current assessments actually measure?
- Why do interactive tasks show larger capability gaps than bounded tasks?
- Can AI evaluation match human judgment quality in structured domain tasks?
- How do automated evaluation metrics differ from human expert judgment?
- Can frontier AI models match expert human performance on specialized tasks?
- Can self-reported AI reliability metrics hide confounding factors like task complexity?
- How does speed of AI search prevent real-time supervision and evaluation?
- Can we measure adaptability and environmental complexity in real-world AI deployments?
- How do researchers benchmark hypothesis-generating systems against each other?
- What distortions do automated benchmarks introduce compared to real tasks?
- Do open source AI models actually match closed models on standard benchmarks?
- How much do current chatbot benchmarks actually measure real model quality?
- How do we measure stages of automation consistently across different domains?
- What specific research-debugging tasks measure AI self-improvement capability?
- Why do AI benchmarks show rapid saturation from near-zero to near-perfect?
- How fast do new benchmarks get adopted across the AI research community?
- How much capability remains hidden in unreleased proprietary models?
- Why do hallucination rates differ between vendor AI products and student-used models?
- Can third-party evaluators embedded in labs measure AI-led R&D work reliably?
- How does capability evaluation differ from alignment evaluation in difficulty?
- Which specific AI R&D tasks does AIDE2 benchmark itself against during selection?
- How much does model publication lag behind actual training completion?
- What would whole-system AGI evaluation look like in practice?
- What baseline rates should AI scheming studies measure behavior against?
- What distinguishes pace controls like evaluators from other governance approaches?