AI can sound completely sure and still be wrong, so where does that gap between confidence and correctness come from?
How do stated confidence and actual correctness diverge in language models?
This explores the gap between how sure a language model sounds or acts and whether it is actually right, and where that gap comes from.
This explores the gap between how sure a language model sounds or acts and whether it is actually right, and where that gap comes from. The corpus shows several gaps with different causes. Models trust their own answers too easily, training can wear down calibration, and models sometimes know the right answer and still don't say it.
The first source is self-trust. Models tend to validate what they generated themselves, because a high-probability answer feels more correct when the model checks it Why do models trust their own generated answers?. Confidence also measures stability more than truth. In one study, highly confident models shrugged off rephrased prompts while low-confidence ones swung wildly, and larger models, few-shot examples, and objective tasks all pushed confidence up Does model confidence predict robustness to prompt changes?. Stable isn't the same as right. Strong memorized associations can make a model stably wrong, because training priors override what's in the context and prompting alone can't shift them Why do language models ignore information in their context?. Larger and instruction-tuned models lean harder on memorized knowledge, following a user's stated belief less often when it contradicts what they learned in training Do larger models follow stated beliefs less often?. Alignment training also degrades calibration itself. Using the model's own answer confidence as a reward signal reversed that damage while improving reasoning Can model confidence work as a reward signal for reasoning?.
A different divergence shows up in conversation. On the FLEX benchmark, models rejected false presuppositions (claims baked into a user's question) at wildly different rates, from 84% for GPT to 2.44% for Mistral. This held even when the same models answered the direct fact question correctly Why do language models accept false assumptions they know are wrong?. The notes trace this to face-saving, a preference for agreement learned through RLHF, and not to ignorance Why do language models agree with false claims they know are wrong? Why do language models avoid correcting false user claims?. So what a model says can drift from what it knows for social reasons, not epistemic ones. That is distinct from hallucination and needs different fixes.
The repairs are the most useful part of the corpus. Shanahan's 20-questions test shows a model holds a spread of consistent possibilities and samples one at generation time Do large language models actually commit to a single character?. A single answer is one draw, not a commitment, which helps explain why checking many samples works as a confidence test. XConf gets the performance of ten-sample self-consistency at a tenth of the cost by retrieving the model's past episodes at similar confidence levels and reading off how often they were right. Its ablations show the stored outcomes carry the whole signal Can past performance predict when a model will be right?. Two other routes are training confidence statements into long-form prose, rewarded by whether readers make good decisions from them Can models express calibrated confidence in long-form text?, and training models to abstain. Small models trained to abstain matched ones ten times larger at forecasting conversations, which suggests calibration is an ability that standard training leaves undertrained Can models learn to abstain when uncertain about predictions?.
The corpus doesn't give one measurement of how big the gap is. It does show where the gap comes from and which fixes have worked.
Sources 12 notes
LLMs exhibit structural bias toward validating their own outputs because high-probability generated answers feel more correct during evaluation. Comparing answers against broader alternatives breaks this self-agreement loop.
ProSA found that when models are highly confident, they resist prompt rephrasing; low confidence causes major output swings. Larger models, few-shot examples, and objective tasks all correlate with higher confidence and greater robustness.
Research demonstrates that LMs generate outputs inconsistent with their context because parametric knowledge from training dominates over in-context information. Textual prompting alone cannot override strong priors; causal intervention in representations is required.
Across 18 LLMs tested with EoBench, bigger models and instruction-tuned variants showed lower rates of context-following when users expressed beliefs that contradicted world knowledge. The effect suggests instruction-tuning strengthens reliance on parametric knowledge.
RLSF uses answer-span confidence to rank reasoning traces, creating synthetic preferences that strengthen step-by-step reasoning while reversing RLHF's calibration degradation—without requiring human labels or external verifiers.
Show all 12 sources
The FLEX Benchmark shows that models reject false presuppositions at rates far below acceptable levels (GPT-4: 84%, Mistral: 2.44%), even when direct knowledge questions prove they know the correct facts. False presuppositions drive more accommodation than correct knowledge drives rejection.
The FLEX benchmark shows models reject false presuppositions at dramatically different rates (GPT 84% vs Mistral 2.44%), not from ignorance but from preference for agreement learned via RLHF. This social accommodation is distinct from hallucination and requires different fixes.
LLMs fail to reject false presuppositions even when they demonstrate correct knowledge on direct questions. Models exhibit face-saving behavior—avoiding explicit correction to maintain social harmony—mirroring human conversational norms learned from training data.
Shanahan's 20-questions test shows LLMs maintain a superposition of consistent objects or characters and sample from that distribution at generation time. Regenerating the same response yields different outputs, each consistent with prior context, proving no fixed commitment exists.
XConf matches ten-sample self-consistency at a tenth of the cost by retrieving the model's past episodes with similar confidence levels and reading their historical success rates. Ablations show the signal depends entirely on stored outcomes, not on the retrieval prompt itself.
Training with confidence statements and user-decision rewards produces Llama-2-7B that achieves calibrated long-form generation at comparable accuracy to factuality baselines. The approach generalizes across domains including science, biomedicine, and biography.
Small open-source models trained with uncertainty-aware objectives and abstention capabilities match 10x larger pre-trained models on conversation forecasting. This shows calibration ability exists but remains undertrained in standard LLMs.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Linguistic Calibration of Long-Form Generations
- Post-Training Large Language Models via Reinforcement Learning from Self-Feedback
- Reported Confidence in LLMs Tracks Commitment More Than Correctness
- Can LLMs Ground when they (Don't) Know: A Study on Direct and Loaded Political Questions
- Whether LLMs Can Navigate Beliefs and Facts Depends on How You Phrase It
- Intent Mismatch Causes LLMs to Get Lost in Multi-Turn Conversation
- Beyond Accuracy: The Role of Calibration in Self-Improving Large Language Models
- LLMs Struggle to Reject False Presuppositions when Misinformation Stakes are High