Can frontier exams really measure cutting-edge AI capability?

When models exceed 90% on popular benchmarks like MMLU, those benchmarks stop measuring anything at the frontier — the ceiling compresses real capability differences into noise. Humanity's Last Exam (HLE) is the explicit response: 3,000 questions across dozens of subjects, built by subject-matter experts, each with an unambiguous verifiable solution that cannot be answered by quick internet retrieval. SOTA models show low accuracy and poor calibration on it, exposing a real gap to the expert human frontier.

Two qualifications make this more than a "harder benchmark" announcement. First, the authors expect rapid saturation again — benchmark history shows models leaping from near-zero to near-perfect quickly, so they anticipate >50% accuracy within a year. Difficulty buys discrimination only temporarily. Second, and more durably interesting: high HLE accuracy would demonstrate expert-level closed-ended knowledge and reasoning, but would not indicate autonomous research or creative open-ended problem-solving. HLE measures structured academic problems, not the open-world capability that actually matters for deployment.

So HLE is best read as the closed-ended counterpart to open-world evaluation. Since Do automated benchmarks hide what frontier AI systems can really do?, the two together bracket the measurement problem: frontier exams restore discrimination on verifiable knowledge, open-world evals capture the messy long-horizon capability that no closed exam can. Neither alone is sufficient, and treating a high exam score as evidence of general capability is exactly the inference the paper warns against.