Reported Confidence in LLMs Tracks Commitment More Than Correctness

Confidence is defined as an estimate of the probability that a chosen answer is correct1–3. Verbal confidence reports are widely used as uncertainty measures in large language models4–6, but it remains unclear whether they are best understood as estimates of correctness at all. We test this using a two-stage abstention paradigm inspired by neuroscientific studies of perceptual decision making7,8: a model first answers a question and reports its confidence, then later decides whether to commit that answer to a user or abstain. Across four non-reasoning models, different prompt framings, and confidence formats, verbal confidence predicted the later commit/abstain decision substantially better than it predicted whether the answer was actually correct. Calibrated token log-probabilities showed a contrasting profile: their ability to predict abstention was closely coupled to their ability to discriminate correctness — the signature of an answer-evidence signal. We then asked whether VC’s behavioural alignment is reducible to its correctness-tracking, by removing the variance in verbal confidence shared with calibrated log-probabilities and examining the residual.

Introduction. An important desideratum for large language models is the production of confidence signals that convey their internal estimate of the correctness of their answer1,2. Such signals are operationally important: they communicate how much weight a user should place on a model’s output, and they may also support the model’s own downstream decisions, such as deferring to a human, seeking additional information, or revising a prior commitment2,3,10–12. Multiple elicitation methods have been proposed, including log-probability-based confidence, sampling-based methods, and verbal confidence — the model’s explicit response to a direct question about how confident it is in its answer4,5,13–15. Verbal confidence is of particular interest because it is a black-box measure, applicable in the many settings where the model’s internal probabilities are not accessible. This makes it the only confidence measure available for many widely deployed reasoning and non-reasoning models, and accordingly the subject of growing interest as a default reliability signal.

Discussion / Conclusion. We asked whether verbal confidence (VC) in large language models is best understood as an estimate of answer correctness or as a signal more closely tied to downstream behavioural control. Across four non-reasoning models, two datasets, two abstention prompt framings, and two confidence formats, the evidence consistently favoured the latter interpretation. VC predicted the model’s subsequent commit/abstain decision substantially more strongly than it predicted objective correctness, whereas calibrated log-probability confidence (Cal-LP), derived from the answer-token distribution, showed a contrasting profile: its prediction of abstention remained closely coupled to its prediction of correctness. These contrasting profiles point to structurally different signals: Cal-LP behaves as a truth-evidence signal whose behavioural relevance is inherited from its correctness-tracking, whereas VC carries an additional decision-aligned component that is not reducible to truth-tracking.