Why do agent benchmarks not predict real economic value?

The puzzle ALE (Agents' Last Exam) starts from is that benchmark victories have accumulated faster than economic transformation: models win at olympiad math, competitive programming, and world-champion games, yet professional deployment stays muted. The paper's claim is that this is not mainly a model problem but an evaluation problem — the field optimizes what it measures, and it has been measuring abstract competence on clean, short tasks rather than the long-horizon, tool-intensive work professional practice requires. So they build a benchmark from work experts have already shipped, anchored to the U.S. federal occupational taxonomy (SOC/O*NET): 55 sub-fields, 13 industry clusters, 960 workflows scored by deterministic checks and rubrics rather than open-ended LLM judging. The hardest tier sits below a 1% full pass rate across mainstream harness/backbone configurations.

This matters because benchmarks are steering instruments, not just scoreboards — they "define engineering targets and often determine which domains become tractable." If the chosen targets are contests, agents get good at contests. The argument convergent-with Do automated benchmarks hide what frontier AI systems can really do? but takes the opposite methodological route: ALE keeps benchmark-scale automation and deterministic scoring rather than retreating to small-sample qualitative study, betting that GDP-relevant tasks can be made verifiable at scale.

The counterargument is the one ALE's own authors anticipate elsewhere in this cluster: difficulty buys discrimination only temporarily. A near-zero pass rate today is exactly the signature that preceded rapid saturation on prior benchmarks. The deeper risk is that deterministic scoring of "economically valuable" workflows still abstracts away the messy human-coordination and judgment work that Does a single benchmark score actually predict agent readiness? identifies as the actual bottleneck — so even a saturated ALE might not certify GDP impact, only a higher grade of the same artifact.