Does verbal confidence actually predict answer correctness?

Verbal confidence — the model's spoken answer to "how sure are you?" — is treated as the default reliability signal precisely because it is black-box: applicable even when internal probabilities are inaccessible, which is most deployed models. The tacit assumption is that it estimates the probability the answer is correct. This paper's two-stage abstention paradigm (answer + report confidence, then later commit-or-abstain) breaks that assumption. Across four non-reasoning models, prompt framings, and confidence formats, verbal confidence predicted the model's own commit/abstain decision substantially better than it predicted objective correctness.

Calibrated token log-probabilities show the opposite profile: their ability to predict abstention is tightly coupled to their ability to discriminate correct answers — the signature of a genuine truth-evidence signal. When the authors regress out the variance verbal confidence shares with log-probs, a residual decision-aligned component remains that is not reducible to truth-tracking. So the two signals are structurally different, not noisy versions of the same thing: log-probs carry an answer-evidence signal; verbal confidence carries an additional behavioral-control signal.

This matters because it undercuts using verbal confidence as a correctness proxy — which means it undercuts using self-reported "how sure are you?" numbers for calibration, deferral, or user trust. It grounds a broader pattern where models' spoken self-reports diverge from what their internal computation encodes: the model reports what it will do with the answer, not how true the answer is. This complicates the persuasion literature too — if models sound sure, that conviction indexes commitment, not evidence.