AI keeps acing its benchmarks — but if those benchmarks don't measure real work, what exactly are we tracking?

This explores whether automated benchmarks capture real progress on long-horizon tasks, and the corpus is unusually blunt: they mostly don't, and the reasons are structural rather than incidental. The sharpest statement comes from work showing that agents which sweep abstract contests still fail real occupational workflows — the so-called benchmark-to-GDP gap, where the field optimizes exactly what it measures and has chosen to measure contests rather than work Why do agent benchmarks not predict real economic value?. The distortion runs both ways: automated benchmarks *overstate* capability by privileging precisely-specified, auto-gradable tasks, and *understate* it by missing emerging capabilities that only show up in open-ended settings Do automated benchmarks hide what frontier AI systems can really do?.

The most direct evidence is temporal. When you actually stretch tasks out, rankings scramble. Models that look interchangeable on single-turn benchmarks diverge dramatically once you run them across 50 round-trip relays — degradation curves that are completely invisible to standard short-interaction tests only become visible around relay 25 Do short benchmarks predict how models perform over long workflows?. And what predicts success over the long haul isn't the thing benchmarks are best at scoring: it's *persistence* through repeated benchmark-edit-incorporate cycles, where most models quit early or burn their budget unproductively regardless of how sharp their initial output was What predicts success in ultra-long-horizon agent tasks?. A one-shot score simply cannot see this.

There's a second, quieter failure mode: benchmarks can be gamed by memorization without anyone noticing. A model that reconstructs half of MATH-500 from partial prompts but scores zero on a post-release benchmark reveals that a chunk of apparent "reasoning progress" was contamination — the benchmark measured recall dressed as capability Does RLVR success on math benchmarks reflect genuine reasoning improvement?. This is the same coin as the GDP gap: a clean-looking number that doesn't correspond to the thing you thought you were measuring.

What's the alternative the corpus points toward? Not abandoning measurement, but changing its texture. Open-world evaluation through qualitative log analysis — reading what the agent actually did, with cost reported honestly — catches capabilities earlier and corrects the distortions Do automated benchmarks hide what frontier AI systems can really do?. Interestingly, some systems already treat empirical benchmarking as an *engine* rather than a report card: the Darwin Gödel Machine self-improves by validating agent variants against real tasks and keeping an evolutionary archive, which is benchmarking used as a discovery loop instead of a scoreboard Can AI systems improve themselves through trial and error?.

The thing you might not have expected: reliability at extreme length turns out to be an *architecture* problem, not a raw-capability one, which is precisely why single-number benchmarks mislead. Million-step tasks can be solved error-free by small non-reasoning models — if you decompose them extremely and vote at each step Can extreme task decomposition enable reliable execution at million-step scale?. A benchmark that scores the base model in isolation would rank it poorly, yet the same model inside the right harness runs flawlessly for a million steps. That inversion — where the harness, the workflow structure, and the persistence budget matter more than the model's benchmark score Can LLMs actually forecast time series better than we think?, Can explicit behavior maps help weaker planners compete with stronger models? — is exactly what an automated long-horizon benchmark is worst positioned to capture.