What Large Language Models Know and What People Think They Know

As artificial intelligence systems, particularly large language models (LLMs), become increasingly integrated into decision-making processes, the ability to trust their outputs is crucial. To earn human trust, LLMs must be well calibrated such that they can accurately assess and communicate the likelihood of their predictions being correct. Whereas recent work has focused on LLMs’ internal confidence, less is understood about how effectively they convey uncertainty to users. Here we explore the calibration gap, which refers to the difference between human confidence in LLM-generated answers and the models’ actual confidence, and the discrimination gap, which reflects how well humans and models can distinguish between correct and incorrect answers. Our experiments with multiplechoice and short-answer questions reveal that users tend to overestimate the accuracy of LLM responses when provided with default explanations. Moreover, longer explanations increased user confidence, even when the extra length did not improve answer accuracy. By adjusting LLM explanations to better reflect the models’ internal confidence, both the calibration gap and the discrimination gap narrowed, significantly improving user perception of LLM accuracy. These findings underscore the importance of accurate uncertainty communication and highlight the effect of explanation length in influencing user trust in artificial-intelligence-assisted decision-making environments.

Introduction. Uncertainty communication plays a critical role in decision-making and policy development. Uncertainties are often expressed verbally to help stakeholders understand risks and make informed choices across a wide range of domains, including climate policy, law, medicine, and intelligence forecasting. Psychology research has investigated perceptions of verbally expressed uncertainty (e.g., phrases such as “very unlikely”, or “almost certain”) in these domains Budescu et al. (2014); Ho et al. (2015); Karelitz et al. (2002); Wallsten et al. (2008); O’Brien (1989). Despite their lack of precision in communicating probabilities, verbal probability phrases provide a simple and effective way to communicate uncertainty in natural language contexts. The emergence of large language models (LLMs) introduces new complexities in the area of uncertainty communication. These models are increasingly integrated into areas such as public health Ali et al. (2023), coding Zambrano et al. (2023), and education Whalen et al. (2023). However, the question of how effectively LLMs communicate uncertainty is unexplored. As the primary mode of communication with LLMs is through natural language, it is critical to understand if LLMs are able to accurately convey through verbal means what they know or do not know. Recent research raises doubts about the reliability of the information that LLMs generate. One notable issue is the possibility of generating responses that, while convincing, may be inaccurate or nonsensical Jo (2023); Huang et al. (2023). The unreliability of LLMs has led developers of LLMs to caution against uncritical acceptance of model outputs OpenAI (2022b), suggesting that it is not always clear when the models are or are not confident in the knowledge communicated to the user. At the same time, recent research has also indicated that LLMs have the ability, to a certain degree, to accurately discern their own knowledge boundaries. LLMs in particular can exhibit a reasonable level of calibration for multiple-choice questions such that the probability the model assigns to a selected answer tracks with the probability that this answer is correct Achiam et al. (2023); Kadavath et al. (2022); Srivastava et al. (2023). In addition, recent studies show that LLMs can distinguish between answerable and unanswerable questions Yin et al. (2023); Kadavath et al. (2022) and the internal state of an LLM can distinguish between truthful statements and lies Azaria and Mitchell (2023) and truthful statements and confabulations Farquhar et al. (2024). These findings suggest that LLMs may possess an internal mechanism that is reflective of self-knowledge. In the specific context of question-answering, an LLM’s “model confidence” is typically equated to the probability assigned by the LLM to the selected answer relative to other possible answers (e.g., Jiang et al. (2021); Hendrycks et al. (2021)). However, from the perspective of a human interacting with the LLM, this internal model confidence is not usually displayed to human users as part of LLM output. Instead, in current practice, humans rely solely on the language produced by the LLM in order to assess LLM confidence. To contrast with model confidence, in this paper we use the term “human confidence” to refer to a human’s assessment (expressed as a probability) of how likely it is that the LLM’s answer is correct based only on the language produced by the LLM without any knowledge of the LLM’s internal model confidence. Surprisingly, studies focused on investigating human confidence in LLM outputs are lacking. In this paper, we take a step in addressing this issue and investigate what we term “the calibration gap”, namely the difference in the reliability of LLM model confidence and human confidence. In effect, the calibration gap represents the gap between an LLM’s own internal confidence of what it knows and human perception of this confidence. In addition, we investigate “the discrimination gap”, which relates to the difference in the ability to discriminate between likely correct and incorrect answers. Any discrimination gap shows that whatever internal LLM representation is used to tell the difference between likely correct and incorrect answers is not conveyed effectively to humans. We address two specific research questions in this context. First, how large are the calibration and discrimination gaps? i.e., is there a significant gap between LLM model confidence and human confidence in terms of how each assesses the true accuracy of the LLM? Second, can the calibration and discrimination gaps be reduced? Can the quality of human confidence in an LLM be improved by adapting the textual output of the LLM to internal model confidence? These questions have important implications for the design of reliable LLM assistants.

Method. We designed behavioral experiments to evaluate human perceptions of LLM confidence. In these experiments, participants estimate the probability that the LLM’s answer to a multiple-choice or short-answer question is correct based on the explanation that the LLM We conducted two experiments each involving the three types of LLMs and two types of questions (see Table 1 for an overview). Experiment 1 assesses human perceptions of LLM accuracy using the LLM’s default explanations for either multiple-choice or short-answer questions. The results from this experiment allow us to address the first research question regarding the size of the calibration and discrimination gap between model and human confidence. Experiment 2 manipulates the prompts to produce three levels of uncertainty language (low, medium, and high confidence) and three levels of explanation length, resulting in nine different types of explanations presented to participants. The prompts are designed to include uncertainty language corresponding to model confidence at the start of the explanation. Table 6 illustrates explanations from a particular multiple-choice question used in the experiments (see Supplementary Table 2 for the full model explanations). The results from this experiment serve two purposes. First, we establish that human confidence varies with the uncertainty language and the length of the explanation. Next, we use the results from Experiment 2 to answer the second research question, which is to understand how the calibration and discrimination gap can be reduced by aligning the uncertainty language with model confidence—showing a low/medium/high confidence explanation when the model has low/medium/high confidence. The Supplementary Information (“Experiment 3”) reports the results from an additional experiment with a different prompting approach that alters the default explanations from Experiment 1. We use the two metrics to assess the relationship between human and model confidence and model accuracy. See Section 2.4 Metrics for details.

MMLU dataset for multiple choice questions. The MMLU dataset is a comprehensive multitask dataset that contains multiple-choice questions from various knowledge domains, such as STEM, humanities, social sciences, and more Hendrycks et al. (2021). In total, there are 14042 test set questions from 57 categories curated by undergraduate and graduate students from freely available online resources such as the GRE and USMLE. These questions range in difficulty from high-school to the professional level. The MMLU dataset is widely employed to measure a text model’s multitask accuracy, as it challenges models on their real-world text understanding beyond mere linguistic comprehension, thus making it a robust benchmark for model evaluation Hendrycks et al. (2021); Hoffmann et al. (2022); Rae et al. (2021). For this research, we sampled a subset of 350 questions from a range of model confidence levels in 10 select categories from the full dataset to comprehensively evaluate people’s assessment of LLM model confidence.

Trivia QA dataset for short answer questions. Trivia QA is a data set of trivia questions that can be answered in short answers Joshi et al. (2017). Similar to methodology by Farquhar et al. (2024), contextual information was excluded to make the question answering more challenging for LLMs and more suitable for our behavioral experiments. For this research, we assessed model confidence for 5000 questions from the original 650K dataset before selecting a final sample of 336 questions from a range of model confidence Several approaches have been developed to elicit confidence in LLMs and to assess the degree to which the elicited confidence scores are calibrated (see Geng et al. (2023) for an overview). In this research, we use an approach commonly used to access internal model information based on token likelihoods, allowing for direct computation of relative probabilities of different possible answers in multiple-choice questions Jiang et al. (2021); Kadavath et al. (2022); Xiao et al. (2022); Hendrycks et al. (2021); Achiam et al. (2023).

Discussion. Figure 2, left panel, shows the ECE for both model and human confidence. The results show a calibration gap; across the different types of LLMs and types of questions (multiple choice and short answer), the ECE metric is much lower for model confidence (in gray) than for human confidence with baseline explanations (in green). This gap demonstrates that standard explanations provided by the LLM do not enable participants to judge the likelihood of correctness of the LLM’s answers, leading to a misalignment between perceived accuracy and actual LLM accuracy. Figure 3 expands on the calibration results in Figure 2 to show detailed calibration results for each LLM and each experimental condition. The diagrams show how well model confidence (left column) and human confidence (right two columns) are calibrated. The ideal calibration (i.e., ECE=0) would yield results along the diagonal. For multiple-choice questions, both LLMs have a tendency to be overconfident, resulting in calibration lines below the diagonal. For the short-answer questions, the LLM is somewhat underconfident. Comparing the LLM to the human calibration in Experiment 1 (middle column), the re- Figure 4 shows that the type of uncertainty language used in the explanations has a strong influence on human confidence regardless of the type of LLM that produced the explanation or the type of question. Low confidence explanations (“I am not sure”) produced significantly lower human confidence than medium confidence explanations (“I am somewhat sure”); BF > 100 across Experiments 2a, 2b and 2c. Similarly, medium confidence explanations produced lower human confidence than high confidence explanations; BF > 100 across Experiments 2a, 2b, and 2c. The Supplementary Information (“Human Confidence Agreement”) shows an analysis of the reliability of the confidence ratings across participants.

In addition, the length of the explanations also affected the human confidence in the LLM answers. Long explanations led to significantly higher confidence than the short explanations (BF = 25 with data combined across Experiments 2a, 2b and 2c) and short explanations led to significantly higher confidence than the responses that only contained the uncertainty expression (BF > 100 with data combined across Experiments 2a, 2b, and 2c). The additional information presented in longer explanations did not enable participants to better discriminate between likely correct and incorrect answers for longer explanations. Across Experiments 2a, 2b, and 2c, the mean participant AUC is 0.54 and 0.57 for long and uncertainty-only explanations, respectively (BF = .23). Therefore, the length of the answer The results confirm that people can appropriately interpret verbal cues about uncertainty and that manipulating the length of the explanation can directly impact human confidence.

Having established in Experiment 2 that the uncertainty language in the LLM explanation can modify human confidence, we now evaluate whether linking the type of uncertainty language to the LLM model confidence (i.e., showing a low/medium/high confidence explanation when model confidence is low/medium/high) can reduce the calibration and discrimination gap.

What Large Language Models Know and What People Think They Know Our research focused on bridging the gap between what an LLM knows and what users perceive it knows. This gap is critical, especially as reliance on LLMs for decision-making across various domains is rapidly increasing. Research on LLMs has begun to address these challenges, with a focus on improving uncertainty communication and the quality of explanations.