Can language models reliably judge their own candidate quality?
LLMs fluently generate candidates across complex spaces but may misestimate their value and uncertainty. Understanding this gap matters for steering AI-driven discovery toward real experimental outcomes rather than internal confidence.
The paper defines scientific discovery as optimizing "expensive-to-evaluate objectives over vast, structured, and open-ended hypothesis spaces" such as molecules, protein sequences, and programs. Its diagnosis is that LLMs give expressive priors over such spaces, but "their likelihoods and self-assessments are unreliable proxies for the objectives and calibrated epistemic uncertainty, especially for novel candidates outside the observed data distribution." The conclusion states the same split as complementary limitations. LLMs have "strong structured generative capabilities and domain priors, but cannot reliably estimate the true value of external objectives or their associated uncertainty." Bayesian optimization "delivers uncertainty-aware experimental decisions from sparse observations, yet struggles to autonomously propose valid candidates in complex discrete spaces."
The Large Discovery Model (LDM) is the proposed division of labor. It is a recurrent architecture in which a generative model proposes and refines candidate designs, while a Bayesian non-parametric reward surrogate (a Gaussian process, per the conclusion) predicts performance and quantifies uncertainty. The result is an "uncertainty-aware value" that guides candidate generation, refinement, and selection. The value signal comes from experimental observations rather than from the LLM's own judgment. A discovery memory and the surrogate are updated as each new observation arrives, which makes the loop "empirically grounded."
The introduction places this against inference-time scaling, which has paid off in mathematics, coding, theorem proving, and games, where solutions are "difficult to find but comparatively easy to evaluate" through verifiers, unit tests, or simulators. AlphaZero-like methods and AlphaEvolve are cited as generation-evaluation-search loops with cheap evaluators. Read together with the abstract, discovery is the case where that cheap evaluator is missing. The surrogate takes its place, and it is fitted to a small number of costly experiments rather than assumed.
This sits beside Do foundation models learn world models or task-specific shortcuts?, which also finds that fluent prediction does not mean a reliable internal model of the target. That note is about physical laws and this one about objective values, so the link is thematic. The LDM setting also overlaps with Can autonomous research pipelines discover AI architectures that AutoML cannot?, since neural-network training is one of LDM's three scenarios. That note lets an LLM drive experiments directly, whereas LDM argues the experiment loop needs an explicit uncertainty model. The paper's stress on real observations over internal knowledge is the same instinct as Why do search agents beat memorized retrieval on hard questions?, applied to experiments instead of search.
The excerpt does not establish that LDM works. It names three scenarios (neural-network training, antibody design, and molecular optimization) but gives no results, baselines, evaluation budgets, or effect sizes. It also does not say how candidates are represented for the surrogate, how the uncertainty-aware value is computed, or how much of any gain comes from the surrogate versus the memory or the recurrence. The claim that LLM self-assessments are unreliable is asserted in the abstract and not demonstrated in the passages provided. What the excerpt supports is the design argument, that generation and valuation are separable jobs and the second should not be left to the generator. Whether that argument holds up is a matter for the full paper's experiments.
Inquiring lines that read this note 27
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
Do confidence signals reliably indicate whether AI models will succeed or fail? How can verification systems keep pace with AI generation?- How do cheap evaluators like verifiers change discovery versus optimization?
- How does the generation-verification gap change as models scale up?
- Can ground truth generation remain a distinctly human responsibility?
- When should discovery systems trust external measurements over learned rankings?
- Do LLMs and human forecasters make better predictions when pooled together?
- How do humans handle verification scope when delegating creation to language models?
- Can models be trained to recognize their own generated text reliably?
- Why do humans and zero-shot LLM judges perform worse than trained detectors?
- Why do LLM-judged tournaments fail to estimate candidate value or uncertainty reliably?
- Can existing fairness audits detect LLM self-preference in hiring systems?
- What happens when one AI model both writes and ranks job applications?
- Does verbal confidence in LLMs track decision-making better than actual correctness?
- Does confidence in LLM outputs systematically outweigh accuracy across languages and contexts?
Related concepts in this collection 3
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
-
Do foundation models learn world models or task-specific shortcuts?
When transformer models predict sequences accurately, are they building genuine world models that capture underlying physics and logic? Or are they exploiting narrow patterns that fail under distribution shift?
parallel finding that fluent generation does not imply a reliable internal model of the target
-
Can autonomous research pipelines discover AI architectures that AutoML cannot?
Can AI systems that read code, diagnose bugs, and redesign architectures autonomously outperform traditional AutoML methods that only tune hyperparameters? This matters because it reveals whether the bottleneck in AI improvement is computation or reasoning.
LLM-driven experimentation on neural-network training, without the explicit surrogate and uncertainty model LDM adds
-
Why do search agents beat memorized retrieval on hard questions?
Deep research agents trained on live web search outperform models fine-tuned on static knowledge. Does real-world RL's advantage come from smarter reasoning, or from bypassing the limitations of memorized facts?
same preference for external empirical feedback over the model's internal knowledge, applied to search
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- Large Discovery Models: Empirically-grounded Model-Based Open-Ended Search
- Self-reflective Uncertainties: Do LLMs Know Their Internal Answer Distribution?
- Can You Trust LLM Judgments? Reliability of LLM-as-a-Judge
- Large Models of What? Mistaking Engineering Achievements for Human Linguistic Agency
- Escaping Model Collapse via Synthetic Data Verification: Near-term Improvements and Long-term Convergence
- LLM-Independent Adaptive RAG: Let the Question Speak for Itself
- Large Language Models Cannot Self-Correct Reasoning Yet
- The Homogenizing Effect of Large Language Models on Human Expression and Thought
Original note title
LLMs propose candidates but cannot reliably estimate their value or uncertainty, so the Large Discovery Model couples them with a Gaussian process surrogate