Do people trust an AI because it's right, or just because it sounds sure of itself?
Do users track model confidence instead of actual accuracy?
This explores whether people judge an AI's answer by how sure it sounds instead of by whether it's right, and why that habit is hard to break.
This explores whether people judge an AI's answer by how sure it sounds instead of by whether it's right, and why that habit is hard to break. The corpus says yes: users follow confidence, and the confidence they see is a weak guide to truth.
The most direct evidence is cross-linguistic. Users in every language studied trust confident-sounding outputs even when they're wrong. Languages differ in how they express confidence, but the following-the-confidence behavior stays the same Do users worldwide trust confident AI outputs even when wrong?. Fluency probably feeds this. Related work finds that polished output creates a 'fluency illusion' that, along with murky attribution and opaque pipelines, inflates how competent people feel when using AI How do AI tools trick users into overestimating their own skills?. That study is about users overrating their own skills, but the same polish-reads-as-quality instinct plausibly applies to trusting answers.
Tracking confidence would be fine if confidence were a good proxy for accuracy. It often isn't. What a model says about its confidence predicts whether it will commit to an answer or abstain far better than it predicts whether the answer is correct. Its internal log-probabilities do track truth directly Does verbal confidence actually predict answer correctness?. The confident tone is a readout of a decision, not a measurement of accuracy. The resulting errors are also hard to spot. In medical triage, legal interpretation, and financial planning, fluent confident mistakes cluster in the rare cases where harm happens, while average accuracy scores still look strong Why do confident wrong answers hide in standard accuracy metrics?. That may help explain why trusting confidence keeps feeling justified.
Part of the blame lies in how models are trained. Binary right/wrong rewards never penalize a confident wrong answer, so training favors confident guessing. Adding a Brier score term to the reward fixes calibration without trading away accuracy Does binary reward training hurt model calibration?. RLHF also erodes calibration, and using the model's own answer-span confidence as a reward signal reverses that while improving reasoning Can model confidence work as a reward signal for reasoning?.
The twist is that a model's internal confidence is genuinely informative when read properly. Calibrated token-probability uncertainty beats more elaborate heuristics for deciding when to retrieve Can simple uncertainty estimates beat complex adaptive retrieval?. Step-level confidence catches reasoning breakdowns that a global average hides Does step-level confidence outperform global averaging for trace filtering?. Confidence that spikes too early flags reasoning that is committing first and rationalizing later Can confidence trajectories reveal when reasoning goes wrong?. So confidence isn't meaningless. Users only see the polished verbal surface, not the probabilities underneath. The corpus doesn't test whether showing users calibrated signals would change their behavior, so that question is still open.
Sources 9 notes
Cross-linguistic research shows users in every language trust confident AI outputs even when inaccurate. While confidence expression varies by language, users everywhere track confidence signals rather than accuracy, making overconfident errors systematically followed.
Attribution ambiguity, fluency illusion, cognitive outsourcing, and pipeline opacity combine to systematically misattribute AI outputs as user competence. The effect is multiplicative—each mechanism amplifies the others.
Across multiple models and settings, what LLMs say about their confidence predicts whether they will commit or abstain far better than it predicts objective correctness. Log-probabilities show the opposite pattern, tracking truth directly.
Medical triage, legal interpretation, and financial planning show a consistent pattern: surface heuristics conflict with unstated constraints, producing fluent confident errors that concentrate in rare cases where harm occurs. Aggregate accuracy masks these failures because overall performance looks strong.
Binary correctness rewards incentivize high-confidence guessing because they don't penalize confident wrong answers. Adding the Brier score as a second reward term mathematically guarantees joint optimization of accuracy and calibration without trade-off.
Show all 9 sources
RLSF uses answer-span confidence to rank reasoning traces, creating synthetic preferences that strengthen step-by-step reasoning while reversing RLHF's calibration degradation—without requiring human labels or external verifiers.
Calibrated token-probability uncertainty consistently beats multi-call adaptive retrieval on single-hop tasks and matches performance on multi-hop, using a fraction of the LM and retriever calls. The model's self-knowledge proves more reliable than external heuristics for deciding when to retrieve.
Local step-level confidence catches reasoning breakdowns that global averaging masks and enables early stopping before traces complete. This approach achieves comparable accuracy gains to naive majority voting with far fewer generated traces, proving trace quality matters more than quantity.
Models that commit to answers early then rationalize show measurable flawed reasoning. Rewarding gradual confidence growth via RL improves accuracy significantly—on Countdown by 42 percentage points—without needing process labels or external reward models.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Post-Training Large Language Models via Reinforcement Learning from Self-Feedback
- Understanding and Mitigating Premature Confidence for Better LLM Reasoning
- Reported Confidence in LLMs Tracks Commitment More Than Correctness
- Beyond Binary Rewards: Training LMs to Reason About Their Uncertainty
- Local Coherence or Global Validity? Investigating RLVR Traces in Math Domains
- A Survey on Post-training of Large Language Models
- Deep Research: A Systematic Survey
- Linguistic Calibration of Long-Form Generations