Can benchmark scores be trusted without knowing their origin?

The paper's starting point is that people who build and study LLMs "need to find relevant evaluations, locate their benchmark datasets and code, and understand the settings behind reported scores." Benchmark Radar is offered as a "living database and search engine" for that job, covering LLM evaluation, agentic and tool-use benchmarks, coding, reasoning, safety, and domain-specific evaluations. Its central design choice, as the abstract describes it, is that it "retains source identities and citations so readers can inspect candidate benchmarks and their evaluation evidence." A score is treated as something to trace back to its origin, not as a standalone number.

The introduction gives the reason. Comparing models "requires benchmarks that expose failures and distinguish capability gains from changes in prompts, data splits, or evaluation procedures," and benchmark scores "inform claims of progress" across scientific, industrial, and decision-support settings. If a score can shift because of the prompt, the split, or the procedure, then a number without its source cannot support a comparison. The system is built to keep the source attached. Daily discovery draws on 37 sources, 13 direct connectors and 24 first-party research and engineering feeds, and feeds a searchable catalog, mentions in model cards and technical reports, and score histories. The audited catalog holds 1,283 source records from 4 benchmark catalogs, with 12,916 numeric observations on 790 of those records.

The nearest notes bear on it from two sides. The paper flags that "lexical matching can miss paraphrases and renamed tasks" and says semantic retrieval will need reviewed relevance judgments. Do vector embeddings actually measure task relevance? is a reason not to assume the semantic route closes that gap, since embedding proximity and task relevance can diverge. That link is this note's inference, not the paper's. Separately, Do frontier LLMs actually explore the full space of valid answers? argues that accuracy alone is an incomplete evaluation. This paper makes a neighboring point about infrastructure: a score alone is incomplete without the evidence behind it. One evaluates models on a missing dimension, and the other builds the layer for finding and reading the evaluations themselves.

The excerpt is a description of a system and its caveats, and it reports no findings from the analyses it names. The abstract says the paper examines "benchmark saturation, adoption trends, and the limits of score comparisons," but the excerpt states no results for any of them. It says plainly that "source records are not a count of distinct underlying tests" and that collection limits, failed requests, missing identifiers, and differing snapshot dates affect coverage. Retrieval precision, task suitability, and time saved "remain to be evaluated," the worked example has "no controlled baseline," and the arXiv route does not backfill earlier papers. Even the authors' own searches for one section missed two relevant earlier studies. What follows at that strength is a design claim: keeping provenance attached makes benchmark evidence inspectable. It is not a demonstrated improvement in finding benchmarks or comparing scores.