Why do readers trust LLM explanations more than they should?

A controlled study comparing an LLM's "model confidence" (the relative probability it internally assigns its selected answer) against "human confidence" (what a reader judges from the model's explanation alone) finds the two are far from aligned. Across two experiments using MMLU multiple-choice questions and TriviaQA short-answer questions, with three types of LLMs, "the ECE metric is much lower for model confidence... than for human confidence with baseline explanations," meaning the model's own probability estimates track its accuracy far better than a reader's impression of the same explanation does. The paper names this the "calibration gap" and pairs it with a "discrimination gap" — the model can tell likely-correct from likely-incorrect answers internally, but when people are given only its explanation, "users tend to overestimate the accuracy of LLM responses when provided with default explanations."

The reason, per the paper, is that the model's internal confidence is never exposed to users in ordinary use — "humans rely solely on the language produced by the LLM in order to assess LLM confidence." That language is poorly matched to the real probability of correctness: multiple-choice explanations run overconfident ("calibration lines below the diagonal") while short-answer explanations run underconfident, and neither failure mode is visible to a reader with no other signal to check against. The gap is a translation failure between an internal scalar (token-likelihood-derived confidence) and a natural-language explanation that was never designed to carry that scalar faithfully.

This sits alongside Do users worldwide trust confident AI outputs even when wrong?, which finds the same overreliance pattern driven by confident-sounding language; this paper adds that even default, non-manipulated explanations produce the gap, not just unusually confident ones. It also gives empirical backing to Does verbal confidence actually predict answer correctness?: if verbal confidence is decision-aligned rather than truth-aligned, a gap between what the words convey and what the log-probabilities actually know is exactly what should follow. It extends Why do language models fail confidently in specialized domains? beyond specialized domains to general-knowledge MMLU and trivia questions, with the human reader as an additional point of failure beyond the model alone. For a concrete mechanism that compounds this gap, see Does longer explanation text actually improve answer evaluation?.

The study is a lab experiment on quiz-style judgment tasks (MMLU, TriviaQA) with three LLM types; it establishes that the gap exists and narrows when explanation wording is deliberately matched to internal confidence, but it does not show how durable that narrowing is across repeated interactions, nor whether the same gap size holds for open-ended tasks with no ground-truth answer to score against. The implication the authors draw is a design one: uncertainty communication in LLM interfaces should be engineered to reflect internal confidence directly, because left to produce its own explanations by default, a model will systematically mislead users about how much to trust it.