SYNTHESIS NOTE
Topics›Novel Architectures›this note

Can language models reliably judge their own candidate quality?

LLMs fluently generate candidates across complex spaces but may misestimate their value and uncertainty. Understanding this gap matters for steering AI-driven discovery toward real experimental outcomes rather than internal confidence.

Synthesis note · 2026-09-25 · sourced from Novel Architectures

The paper defines scientific discovery as optimizing "expensive-to-evaluate objectives over vast, structured, and open-ended hypothesis spaces" such as molecules, protein sequences, and programs. Its diagnosis is that LLMs give expressive priors over such spaces, but "their likelihoods and self-assessments are unreliable proxies for the objectives and calibrated epistemic uncertainty, especially for novel candidates outside the observed data distribution." The conclusion states the same split as complementary limitations. LLMs have "strong structured generative capabilities and domain priors, but cannot reliably estimate the true value of external objectives or their associated uncertainty." Bayesian optimization "delivers uncertainty-aware experimental decisions from sparse observations, yet struggles to autonomously propose valid candidates in complex discrete spaces."

The Large Discovery Model (LDM) is the proposed division of labor. It is a recurrent architecture in which a generative model proposes and refines candidate designs, while a Bayesian non-parametric reward surrogate (a Gaussian process, per the conclusion) predicts performance and quantifies uncertainty. The result is an "uncertainty-aware value" that guides candidate generation, refinement, and selection. The value signal comes from experimental observations rather than from the LLM's own judgment. A discovery memory and the surrogate are updated as each new observation arrives, which makes the loop "empirically grounded."

The introduction places this against inference-time scaling, which has paid off in mathematics, coding, theorem proving, and games, where solutions are "difficult to find but comparatively easy to evaluate" through verifiers, unit tests, or simulators. AlphaZero-like methods and AlphaEvolve are cited as generation-evaluation-search loops with cheap evaluators. Read together with the abstract, discovery is the case where that cheap evaluator is missing. The surrogate takes its place, and it is fitted to a small number of costly experiments rather than assumed.

This sits beside Do foundation models learn world models or task-specific shortcuts?, which also finds that fluent prediction does not mean a reliable internal model of the target. That note is about physical laws and this one about objective values, so the link is thematic. The LDM setting also overlaps with Can autonomous research pipelines discover AI architectures that AutoML cannot?, since neural-network training is one of LDM's three scenarios. That note lets an LLM drive experiments directly, whereas LDM argues the experiment loop needs an explicit uncertainty model. The paper's stress on real observations over internal knowledge is the same instinct as Why do search agents beat memorized retrieval on hard questions?, applied to experiments instead of search.

The excerpt does not establish that LDM works. It names three scenarios (neural-network training, antibody design, and molecular optimization) but gives no results, baselines, evaluation budgets, or effect sizes. It also does not say how candidates are represented for the surrogate, how the uncertainty-aware value is computed, or how much of any gain comes from the surrogate versus the memory or the recurrence. The claim that LLM self-assessments are unreliable is asserted in the abstract and not demonstrated in the passages provided. What the excerpt supports is the design argument, that generation and valuation are separable jobs and the second should not be left to the generator. Whether that argument holds up is a matter for the full paper's experiments.

Inquiring lines that read this note 27

This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.

Do confidence signals reliably indicate whether AI models will succeed or fail? How can verification systems keep pace with AI generation? Do LLMs lack causal understanding needed for accurate prediction? Do language models rely on encoded knowledge or primarily on surface patterns? What biases affect LLM judges and how can they be reduced? Can employers detect AI-assisted applications and should they be permitted? Why do independent language models converge on similar outputs? Why do NLP benchmarks treat human disagreement as noise rather than signal? How can we verify and preserve mathematical understanding in AI-generated proofs? Can external verification replace human reasoning oversight for correctness? How should persistent agent-authored code be governed and shared? How aligned are models' verbal explanations with their actual internal reasoning? Can AI research automation improve discovery pace without sacrificing quality? Why does model self-revision increase confidence in incorrect answers? When does AI explainability improve versus hinder human oversight and trust? How do language models handle contested beliefs versus facts? When do thinking tokens stop improving reasoning performance? Do AI shopping agents exhibit systematic bias in recommendations? Does alignment training modify underlying model representations or just output compliance?

Related concepts in this collection 3

This note in its neighbourhood — explore the map, then jump to a related concept in the list below.

Concept map
15 direct connections · 145 in 2-hop network ·dense cluster Open in graph ↗

Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph

your link semantically near linked from elsewhere

Related papers in this collection 8

Papers most semantically related to this note, ranked by cosine similarity in the embedding space.

Original note title

LLMs propose candidates but cannot reliably estimate their value or uncertainty, so the Large Discovery Model couples them with a Gaussian process surrogate