Nearly every LLM benchmark in a 445-paper review fell short on measuring what its name promises: how should researchers document those gaps?

This explores what researchers should write down about the limits of their benchmarks, so a reader knows how much weight a score can bear. The corpus has no ready-made documentation template. It does have something more useful: a map of where the gaps usually hide, which gives you a checklist. The baseline is grim. A review of 445 LLM benchmark papers found that almost all of them fall short on construct validity, the question of whether a test measures the thing it's named after Do LLM benchmarks actually measure what they claim to measure?. The recurring problems point straight at what to document. Define the ability you're testing. Explain why a task borrowed from older work still fits. Report statistical tests. Say plainly which claims your scores support and which they don't.

The least obvious gap is what you threw away. Benchmarks routinely drop examples where human annotators disagreed, which seems like good data hygiene. But that filtering removed exactly the cases that expose LLM weakness at handling ambiguity. On ambiguous examples, accuracy was 32%, against 90% on the cleaned versions, and standard evaluation never shows the difference Do standard NLP benchmarks hide LLM ambiguity failures?. So a validity statement should list the curation choices and what they excluded, not just what's left. The same goes for scoring choices. LR²Bench scores only final answers against fixed correct answers, because grading the reasoning steps would reward models that imitate the style of reasoning without getting anything right Should reasoning benchmarks score final answers or reasoning traces?. Each choice like that is a validity decision, and it should be written down with its reason.

Contamination is the gap readers care about most and authors document least. One model could rebuild 54.6% of the MATH-500 problems from partial prompts, yet scored 0% on a newer benchmark released after its training data was collected Does RLVR success on math benchmarks reflect genuine reasoning improvement?. A companion note adds a twist: a training method can genuinely change how a model reasons while the benchmark gains still come from memorization. The two happen at different levels of measurement Can genuine reasoning activation coexist with contaminated benchmarks?. A good report says which one its scores can speak to. Running the same test on data released after the model's training cutoff is the cheapest check available.

A second idea in the corpus is that documentation shouldn't be prose alone. It can travel with the score as evidence. Benchmark Radar keeps each score linked to its source and original setup, because a number with no record of where it came from can't support fair comparisons between models Can benchmark scores be trusted without knowing their origin?. BenchShield goes further for agent benchmarks. Benchmark operators can claim an agent "validly completed the task" based on recorded logs of how it ran, not just a final pass/fail number Can infrastructure evidence replace terminal scores in benchmark validation?. That matters because switching to interactive, multi-step evaluation doesn't remove these problems. Comparability, reproducibility, and how evidence maps to a verdict all show up again across the whole sequence of an agent's actions, and they need shared protocols to stay interpretable Do interactive evaluations actually solve the benchmark comparison problem?.

The idea worth borrowing comes from AI paper-writing systems. Spark-to-Paper requires the system to specify what evidence would count before it sees any results Can separating judgment from verification improve research paper reliability?. That is essentially preregistration, the practice of committing to an analysis plan in advance. Applied to benchmarks, it would mean writing down before any testing what the benchmark is meant to measure, what result would count as failure, and what the data excludes. That's the honest version of a validity section. Without it, limitations tend to get written after the scores are in, to fit whatever the scores turned out to be.