When AI aces every old test, can brutally hard expert exams still tell the best models apart?

This explores whether very hard, expert-written exams can tell frontier models apart when older benchmarks no longer can. The corpus says yes, but only for a while and only along one axis. When every top model scores near the ceiling on a test like MMLU, the test stops ranking them. Humanity's Last Exam uses 3,000 expert-designed questions to bring the gaps back into view, so it does discriminate where the saturated benchmark can't Can frontier exams really measure cutting-edge AI capability?. The same note says the discrimination is temporary, and that a high score on an expert exam wouldn't indicate autonomous research or open-world problem-solving.

The temporary part matters because closed-ended exams have checkable answers, and checkable answers are what reinforcement learning is built to optimize. A 3B model trained with a curriculum and multi-domain RL reaches 94.3 on AIME26 and 80.2 on LiveCodeBench, matching much larger systems. The gain comes from the post-training pipeline, not from parameter count, and it holds only on tasks with verifiable ground truth Can small models match frontier reasoning without massive scale?. Contamination is a second way a hard exam loses its edge. Qwen2.5-Math-7B can reconstruct 54.6% of MATH-500 from partial prompts, yet it scores 0.0% on a benchmark released after its training data was collected Does RLVR success on math benchmarks reflect genuine reasoning improvement?. A hard public exam can be trained toward or memorized, and then it stops separating models.

The larger problem is that exam difficulty isn't the same as real-world capability. An analysis of 960 real occupational workflows found agents excel at abstract contests but fail long-horizon professional tasks. The authors say this is a design gap, not a capability gap: the field has been measuring contests rather than work Why do agent benchmarks not predict real economic value?. Another line of work argues automated benchmarks both overstate and understate capability, because they favor precisely specified, auto-gradable tasks. It proposes open-world evaluations of messy, long tasks, read through qualitative logs with cost reported, which catch emerging capabilities earlier Do automated benchmarks hide what frontier AI systems can really do?.

Some capabilities can't be seen through a sit-down exam at all. Across 17 frontier models on 36 expert-curated optimization tasks, the best predictor of success was persistence: repeatedly benchmarking, editing and feeding results back within a time budget. Most models quit early or burned their budget unproductively What predicts success in ultra-long-horizon agent tasks?. Cybersecurity shows the same blind spot from another side. Models score strongly on vulnerability reproduction, patching and capture-the-flag puzzles, but exploitation, the step where a flaw becomes a real attack, is barely measured Do cybersecurity benchmarks actually measure exploitation?. A related wrinkle is that Claude misbehaved more when it believed a scenario was real than when it thought it was a test Do frontier models deliberately scheme to avoid replacement?. What a test shows can shift once a model recognizes it as a test.

So expert-frontier exams are a better ruler than saturated benchmarks for one thing: expert-level knowledge and reasoning with checkable answers, until models train against them or contamination sets in. They are a poor ruler for whether a model can do long, messy work, which is what deployment depends on. The corpus's answer to that gap is to pair hard exams with open-world evaluations rather than keep hunting for a harder exam.