SYNTHESIS NOTE
Topics›RLVR›this note

Can model confidence alone replace external answer verification?

Can LLMs use their own certainty signals instead of external verifiers to improve reasoning? This matters for scaling beyond domains where correct answers can be automatically checked.

Synthesis note · 2026-02-22 · sourced from RLVR
How do domain training techniques actually reshape model behavior? How do you navigate synthesis across fragmented research topics? What does reward learning actually do to model reasoning?

RLVR's reliance on domain-specific verifiers confines it to math and code. Two complementary approaches extend RLVR to general domains by replacing external verification with intrinsic signals.

RLPR (Reinforcement Learning with Reference Probability Reward) uses the LLM's own token probability of generating a reference answer as the reward signal. The probability reflects how well the reasoning process leads to the correct answer and measures how likely the model is to take the correct action. Two key innovations: (1) a Probability-based Reward computed from average decoding probabilities of reference answer tokens, showing better robustness than naive sequence likelihood, and (2) stabilization methods to address the high variance inherent in probability-based rewards. RLPR consistently improves reasoning across Gemma, Llama, and Qwen models on both general-domain and mathematical benchmarks.

INTUITOR goes further: it uses the model's own confidence — self-certainty measured as average KL divergence between the output distribution and a uniform distribution — as its sole reward signal. No reference answers, no external verifiers, no labeled data. The approach is simple: replace the verifiable reward in GRPO with self-certainty scores. The mechanism builds on the observation that LLMs exhibit lower confidence on difficult problems; optimizing for confidence should drive the model toward more reliable reasoning.

Both approaches raise the same fundamental question for future AI: as models develop capabilities beyond human evaluation, self-generated signals may be the only viable training pathway. Since Can model confidence work as a reward signal for reasoning?, there is convergent evidence that intrinsic confidence signals can serve dual roles — improving both performance and reliability.

Since Can reasoning improvement work without answer verification?, RLPR and INTUITOR represent the next step: progressively weaker assumptions about what external signal is needed, from reference verification to reference probability to pure self-certainty.

Inquiring lines that read this note 51

This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.

How do evaluation biases undermine LLM quality assessment systems? Why does verification consistently lag behind AI generation? Does self-reflection enable models to reliably correct their errors? How do language models inherit human biases from training data? Can model confidence signals reliably improve reasoning quality and calibration? Why does self-revision increase model confidence while degrading accuracy? How should we design LLM systems to maintain alignment and control? How effectively do deterministic tools improve language model reasoning on formal tasks? What structural advantages do diffusion language models offer over autoregressive methods? Why do language models reinforce false assumptions instead of correcting them? What properties determine whether reward signals teach genuine reasoning? Do accurate-looking LLM outputs hide structural failures in learning and reasoning? How do multi-agent systems achieve genuine cooperation and reasoning? How does test-time aggregation affect reasoning correctness and reliability? How faithfully do LLMs reflect their actual reasoning in outputs and explanations? Does tokenized intelligence retain genuine value through exchange-based systems? Do language models develop causal world models or rely on statistical patterns? How do adversarial and manipulative prompts attack reasoning models? How can models identify insufficient information and respond appropriately without guessing? How should models express uncertainty rather than forced confident answers? Do harness improvements transfer across model scales or memorize shortcuts? What constrains reinforcement learning's ability to expand model reasoning?

Related concepts in this collection 4

This note in its neighbourhood — explore the map, then jump to a related concept in the list below.

Concept map
14 direct connections · 106 in 2-hop network ·medium cluster Open in graph ↗
Can model confidence alone replace external answ… Can model confidence work as a reward signal for r… Can reasoning improvement work without answer veri… Does self-consistency reliably reward correct answ… What limits how much models can improve themselves…

Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph

your link semantically near linked from elsewhere

Related papers in this collection 8

Papers most semantically related to this note, ranked by cosine similarity in the embedding space.

Search by related questions 5

Suggested questions this note speaks to — click to search the collection, or type your own.

Does optimizing for model confidence actually improve both performance and calibration simultaneously?How do probability-based rewards compare to self-consistency as training signals for reasoning?What happens when confident wrong answers become more rewarded than uncertain correct ones?At what capability level does the generation-verification gap make intrinsic rewards insufficient?Can unsupervised confidence-based training scale to domains beyond human evaluation reach?
Original note title

llm intrinsic probability of generating a correct answer can replace external verifiers as reward signal — extending rlvr to general domains