Benchmark Radar: A Living Database and Search Engine for AI Benchmarks and Evaluation

Benchmark researchers and developers of large language models (LLMs) and other AI systems need to find relevant evaluations, locate their benchmark datasets and code, and understand the settings behind reported scores. We present Benchmark Radar, a living database and search engine for retrieval and discovery of AI benchmarks, covering LLM evaluation, agentic and tool-use benchmarks, coding, reasoning, safety, and domain-specific evaluations. The system combines daily discovery of benchmark papers, repositories, datasets, and releases with a searchable benchmark catalog, mentions in model cards and technical reports, and score histories. It retains source identities and citations so readers can inspect candidate benchmarks and their evaluation evidence. Daily discovery draws on 37 sources: 13 direct connectors and 24 first-party research and engineering feeds. The catalog contains 1,283 source records drawn from 4 benchmark catalogs and 12,916 numeric observations on 790 records. We describe collection and retrieval, audit the full catalog, and examine benchmark saturation, adoption trends, and the limits of score comparisons.

Introduction. The Transformer architecture established attention-based sequence modeling as a foundation for modern large language models (LLMs) [66]. GPT-6 Astra and Claude Fable 5.1 are designed for coding, research, and tasks that span multiple tools [3, 46]. Comparing these models requires benchmarks that expose failures and distinguish capability gains from changes in prompts, data splits, or evaluation procedures [21, 49, 52, 57]. Evaluation also matters beyond general-purpose chat. In recommender systems, Transformer-style sequence models and LLM-based methods support user, item, and preference modeling, with efficient training needed at production scale [29, 55, 63]. Foundation models are also used to model biological, geoscientific, and physical processes [13, 56, 65, 70, 78]. In social and political analysis, language models support text annotation and simulated human samples, while network methods quantify balance and polarization dynamics [4, 16, 35, 36, 45, 58]. Benchmark scores inform claims of progress across these scientific, industrial, and decision-support settings.

Discussion / Conclusion. The census describes this catalog at its recorded cutoff. Collection limits, failed requests, missing identifiers, and different snapshot dates affect its coverage. Source records are not a count of distinct underlying tests. Broader coverage requires additional source collection and review of the evidence already present. Retrieval precision, task suitability, and time saved remain to be evaluated. The worked example combines local queries with web search and has no controlled baseline. Lexical matching can miss paraphrases and renamed tasks. Evaluating semantic retrieval [51] will require reviewed relevance judgments across queries and candidate records, including less familiar benchmark families. The arXiv discovery route collects new submissions; it does not backfill papers first posted before the collection window when later versions appear. The searches used for Section 2 missed two relevant earlier studies [1, 76], which a coauthor identified and read.