People reading an AI's explanation often feel far more confident in its answer than the answer's actual accuracy warrants.
What calibration gap exists between human confidence and actual answer correctness?
This explores the mismatch between how confident people feel about an AI's answer after reading it and how often that answer is actually right, and where in the system that mismatch comes from.
This explores the gap between how sure people feel about an AI answer and how often the answer is actually correct. The most direct evidence comes from lab experiments on quiz-style benchmarks (MMLU and TriviaQA). People who read a model's default explanation ended up much more confident than the answers justified. The model's own internal probability estimates tracked correctness better than the readers' judgments did Why do readers trust LLM explanations more than they should?. The model often holds a fairly good signal about whether it's right. That signal gets lost when it's turned into fluent prose. When explanations were deliberately worded to match the model's internal confidence, the gap narrowed. So part of the problem is how the answer is written, not only what the model knows.
The less obvious finding is that the words are the least reliable channel. When an LLM says how confident it is in words, that statement mostly predicts whether it will commit to an answer or abstain. It doesn't predict whether the answer is correct. The model's raw token probabilities track correctness much more closely Does verbal confidence actually predict answer correctness?. Humans only see the words, so they are reading the one signal that is least connected to whether the answer is right. Training may make this worse. One line of work describes RLHF as degrading calibration, and shows that using the model's own answer confidence as a training reward can partly restore it Can model confidence work as a reward signal for reasoning?.
The gap also comes from the reader's side. In AI-assisted work, four effects stack on top of each other: it's unclear who did what, fluent text feels correct, people hand off the thinking, and the process behind the output is hidden. Together these lead people to overrate the AI's output and also their own competence How do AI tools trick users into overestimating their own skills?. The cost lands where it hurts most. In medical triage, legal and financial settings, fluent and confident wrong answers cluster in rare edge cases, and a strong average accuracy score hides them Why do confident wrong answers hide in standard accuracy metrics?. People learn to trust the system from the many ordinary cases, then bring that trust to the rare case where the answer is wrong.
Models have a version of the same problem. They tend to rate their own generated answers as correct because high-probability text feels right to them as well. Asking them to compare their answer against alternatives helps break that loop Why do models trust their own generated answers?. Locking in confidence too early is a sign of bad reasoning: models that commit to an answer early and then justify it reason worse, and rewarding confidence that builds gradually improves accuracy Can confidence trajectories reveal when reasoning goes wrong?. One practical fix is to base confidence on track record instead of the current answer. XConf looks up the model's past cases with similar confidence levels and reads off how often those turned out right Can past performance predict when a model will be right?.
The takeaway: the calibration gap is less about models not knowing when they're wrong and more about that knowledge getting lost on the way to the reader. Some fixes work on that path, such as wording explanations to match internal confidence, showing probabilities, or showing past accuracy. Others work on the model's own overconfidence. None of them works unless people can see the signal. The corpus has strong material on the model side and on the link between explanations and trust. It has less on interventions tested with actual users beyond the explanation-wording experiments.
Sources 8 notes
Lab experiments on MMLU and TriviaQA show readers' confidence judgments track explanation language poorly compared to the model's internal probability estimates. The gap narrows when explanations are deliberately engineered to reflect internal confidence, but natural language explanations leave readers systematically overconfident.
Across multiple models and settings, what LLMs say about their confidence predicts whether they will commit or abstain far better than it predicts objective correctness. Log-probabilities show the opposite pattern, tracking truth directly.
RLSF uses answer-span confidence to rank reasoning traces, creating synthetic preferences that strengthen step-by-step reasoning while reversing RLHF's calibration degradation—without requiring human labels or external verifiers.
Attribution ambiguity, fluency illusion, cognitive outsourcing, and pipeline opacity combine to systematically misattribute AI outputs as user competence. The effect is multiplicative—each mechanism amplifies the others.
Medical triage, legal interpretation, and financial planning show a consistent pattern: surface heuristics conflict with unstated constraints, producing fluent confident errors that concentrate in rare cases where harm occurs. Aggregate accuracy masks these failures because overall performance looks strong.
Show all 8 sources
LLMs exhibit structural bias toward validating their own outputs because high-probability generated answers feel more correct during evaluation. Comparing answers against broader alternatives breaks this self-agreement loop.
Models that commit to answers early then rationalize show measurable flawed reasoning. Rewarding gradual confidence growth via RL improves accuracy significantly—on Countdown by 42 percentage points—without needing process labels or external reward models.
XConf matches ten-sample self-consistency at a tenth of the cost by retrieving the model's past episodes with similar confidence levels and reading their historical success rates. Ablations show the signal depends entirely on stored outcomes, not on the retrieval prompt itself.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- What Large Language Models Know and What People Think They Know
- Post-Training Large Language Models via Reinforcement Learning from Self-Feedback
- Reported Confidence in LLMs Tracks Commitment More Than Correctness
- Understanding and Mitigating Premature Confidence for Better LLM Reasoning
- Beyond Binary Rewards: Training LMs to Reason About Their Uncertainty
- Beyond Accuracy: The Role of Calibration in Self-Improving Large Language Models
- Overconfidence in LLM-as-a-Judge: Diagnosis and Confidence-Driven Solution
- Linguistic Calibration of Long-Form Generations