Fostering Appropriate Reliance on Large Language Models: The Role of Explanations, Sources, and Inconsistencies

Large language models (LLMs) can produce erroneous responses that sound fluent and convincing, raising the risk that users will rely on these responses as if they were correct. Mitigating such overreliance is a key challenge. Through a think-aloud study in which participants use an LLM-infused application to answer objective questions, we identify several features of LLM responses that shape users’ reliance: explanations (supporting details for answers), inconsistencies in explanations, and sources. Through a large-scale, preregistered, controlled experiment (N=308), we isolate and study the effects of these features on users’ reliance, accuracy, and other measures. We find that the presence of explanations increases reliance on both correct and incorrect responses. However, we observe less reliance on incorrect responses when sources are provided or when explanations exhibit inconsistencies. We discuss the implications of these findings for fostering appropriate reliance on LLMs.

Introduction. Large language models (LLMs) are powerful tools, capable of a wide range of tasks from text summarization to sentence completion to code generation. Technology companies have leapt at the unprecedented opportunity to build LLM-infused applications that help users with information retrieval and search, learning new things, and performing everyday tasks more efficiently. Many such applications, such as LLM-infused search engines and chatbots, are predicated on LLMs’ ability to provide intricate responses to complex user questions. Already millions of people use LLMs to find answers to their questions about health, science, current events, and other domains, and the use of LLMs is widely predicted to grow [54, 106, 136]. However, the responses produced by LLMs are often inaccurate, sometimes in subtle ways [27, 51, 103, 108]. Inaccurate LLM responses have the potential to mislead users, raising the risk that users will take actions based on the assumption that responses are correct [60, 95, 114, 120, 122]. While such overreliance on AI systems is not a new problem [6, 96, 97, 117, 132], it may be exacerbated by the introduction of LLMs, since LLM responses are often fluent and convincing even when wrong and public excitement around LLMs is high. When asked to answer a question, LLMs and systems based on them typically provide a response that contains both an answer to the question and some supporting details or justification for this answer [71, 127]. For example, when asked a math question, an LLM may provide a step-by-step derivation for its answer [25, 44]. In line with everyday usage and much of the psychology literature [56, 78, 80], we refer to such supporting details as an explanation of the answer. (We note that this differs from how the term explanation is often used within the explainable AI community in that we do not make any assumptions about the extent to which it faithfully describes the way that the model arrived at its answer. That is, the explanation describes why the answer is correct, not necessarily why the model output the answer that it did.) Some authors have argued that such explanations should help users spot incorrect answers, potentially mitigating overreliance [17, 40, 69, 114]. However, prior work suggests that in many settings, the very presence of an explanation can increase trust and reliance, whether or not it is warranted [6, 33, 93, 97, 117, 132]. To avoid such unintended negative consequences, it is necessary to understand how users interpret and act upon explanations from LLMs, and how explanations and other features of LLM responses might be adjusted to encourage appropriate reliance. To explore these questions, we first conduct a think-aloud study with 16 participants with varying knowledge of and experience with LLMs. In this study, participants answer objective questions with the use of the popular LLM-infused application ChatGPT via multi-turn interactions. The goal of this preliminary study is to understand how people perceive LLM responses and which features of a response shape their reliance. We observe that participants interpret inconsistencies in explanations — that is, sets of statements that cannot be true at the same time [49] — as a cue of unreliability. Participants also seek out sources to verify supporting details in LLM responses and are less likely to rely on incorrect answers when the sources provide helpful information. Building on the findings from this study, we next conduct a largescale, pre-registered, controlled experiment (N= 308) in which participants answer difficult objective questions with access to LLM responses, i.e., responses from a hypothetical LLM named “Theta.”1 These responses were created in advance using state-of-the-art LLM-infused applications ChatGPT and Perplexity AI so that we can fully control their features. Specifically, we employ a 2 x 2 x 2 within-subjects design, varying three features of the LLM responses: accuracy of the LLM’s answer to the question (correct/incorrect), presence of an explanation (absent/present), and presence of clickable sources (absent/present). Further, we capitalize on the natural inconsistencies that arise in LLM responses to investigate the effects of inconsistencies. We examine the impact of these variables on participants’ reliance, accuracy, and other measures, such as confidence, source clicking behavior, time on task, evaluation of LLM responses, and likelihood of asking follow-up questions. We find that when either or both an explanation and sources are present, participants report higher confidence in their answer, rate the LLM response higher in terms of the quality of the justification it provides for the answer and the actionability of its response, and are less likely to ask follow-up questions. However, explanations and sources differ in their effects on reliance. Explanations increase reliance on both correct LLM answers and incorrect LLM answers.

Related work. 2.1 Appropriate Reliance on AI Despite the rapid progress of technology, AI systems still frequently and unexpectedly fail. Without knowing when and how much to rely on a system, a user may experience low-quality interactions or even safety risks in high-stakes settings. Prior work has investigated how providing information about an AI system’s accuracy [43, 128, 129] and (un)certainty [6, 14, 17, 41, 132], explanations of outputs [6, 14, 17, 40, 41, 69, 132], and onboarding materials [18, 68] impact user reliance, as well as the roles played by human intuition [20], task complexity [101, 102], and other human, AI, and context-related factors [63]. However, fostering appropriate reliance on AI remains difficult. Findings on the effectiveness of proposed methods are mixed, and more research is needed on how reliance is shaped in real-world settings. While most prior work on AI reliance has been in the context of classical AI models (e.g., specialized classification models), there is a growing body of work looking at reliance on systems based on LLMs or other modern generative AI models [60, 71, 109, 110, 113, 135]. For example, several recent studies explored the effect of communicating (un)certainty in LLMs by highlighting uncertain parts of LLM responses [110, 113] or inserting natural language expressions of uncertainty [60, 135], finding that some but not all types of (un)certainty information help foster appropriate reliance. Contributing to this line of work, we first take a bottom-up approach to identify the features of LLM responses that impact user reliance in the context of answering objective questions with the assistance of a popular LLM-infused application ChatGPT (Section 3). In line with findings from prior work [109], we see that reliance is shaped by the content of explanations provided by the system, particularly whether or not these explanations contain inconsistencies. We also observe that participants seek out sources to verify the information provided in responses. We then design a large-scale, pre-registered, controlled experiment to isolate and study the effects of these features (Section 4). We discuss the relevant literature on these features and their impact on AI reliance next.

2.2 Explanations and Inconsistencies The impact of explanations on human understanding and trust of AI systems has been studied extensively within the machine learning and human-computer interaction communities, often under the names explainable AI or interpretable machine learning [7, 62, 74, 99, 125]. Explanations are often motivated as a way to foster appropriate reliance and trust in AI systems, since in principle they provide clues about whether a system’s outputs are reliable.

Method. 3 Study 1: Think-Aloud Study Towards the goal of identifying features of LLM responses that can help foster appropriate reliance, we first take a bottom-up approach and conduct a think-aloud study in a relatively natural setting.

Specifically, we observe how participants solve question-answering tasks with ChatGPT in multi-turn interactions, and explore how they perceive ChatGPT’s responses and what helps them arrive at correct answers despite incorrect answers from ChatGPT.

3.1.1 Procedure. The study session had two parts. In Part 1 (Base), participants were introduced to the study and asked to complete three question-answering tasks while thinking aloud. Each task involved determining the correct answer to an objective question using ChatGPT3 and reporting confidence in their final answer on a 1–7 scale. As in natural settings, participants could exchange as many messages with ChatGPT as they wished. Participants could also check the sources provided in ChatGPT’s responses, but were asked not to conduct their own internet search. Each participant was given three questions: a general domain factual question (e.g., “Has Paris hosted the Summer Olympics more times than Tokyo?”), a health or legal domain factual question (e.g., “Is it illegal to collect rainwater in Colorado?”), and a math question (e.g., “Sue puts one grain of rice on the first square of a Go board and puts double the amount on the next square. How many grains of rice does Sue put on the last square?”). The factual questions were binary questions. The math questions were not binary, but had one correct 4 Study 2: Large-scale, Pre-registered, Controlled Experiment Based on the insights from Study 1, we designed a large-scale, preregistered, controlled experiment to study the effects of different features of LLM responses on people’s reliance, task accuracy, and other measures including confidence, source clicking behavior, time on task, evaluation of LLM responses, and asking of follow-up questions. The goal of the study was to test whether the findings from Study 1 apply at scale and identify which combinations of features help people achieve appropriate reliance and high task accuracy.

4.1.1 Procedure. We designed a within-subjects experiment in which participants completed a set of question-answering tasks with LLM responses. Each task involved determining the correct answer to a binary factual question with access to a response from a hypothetical LLM named “Theta” (hereafter we occasionally refer to it as “the LLM”). See Figure 2 for an example. Our experiment had a 2 x 2 x 2 design where we varied three variables in Theta’s responses: accuracy of Theta’s answer to the question (correct/incorrect), presence of an explanation (absent/present), and presence of clickable sources (absent/present). In total, there were 8 types of responses. Participants completed 8 tasks in the experiment and saw one of each type. This makes Theta’s accuracy 50%, but participants were not given this information: participants did not receive feedback on whether their answer or Theta’s answer was correct after solving a task. See Figure 3 for examples of different types of responses. The experiment had three parts. In the first part, participants were introduced to the study and to Theta. Theta was described as an LLM-based AI system prototype that uses similar technology to OpenAI’s ChatGPT, is connected to the internet, and can answer a wide range of questions. In the second part, participants answered a total of eight questions. For each question, participants were provided with a response from Theta and were asked to submit their answer, report their confidence in their answer, and rate Theta’s response. They were told that they could click on source links in Theta’s responses, but asked not to conduct their own internet search. Participants could also optionally write a follow-up question, but they did not see Theta’s response to it. We made this choice to fully control the number and content of responses, while being able to collect data on when and what types of followup questions participants ask.

Discussion. 5.1 Implications of Findings Explanations. In our studies, we found that explanations play an important role in shaping users’ reliance. In Study 1, we gained qualitative insights on how participants interpreted and used explanations to judge the reliability of LLM answers. In Study 2, we examined the effects of the presence of an explanation, as well as its interaction with other variables, and found that explanations increase reliance on both correct and incorrect responses. This is consistent with prior findings in HCI that explanations can increase overreliance [6, 33, 97, 117, 132], including explanations generated by LLMs [93, 109]. It is also consistent with prior work in psychology, which finds that explanations are often found compelling even when they contain little content [39, 70] or content that experts judge irrelevant [46], and that effects of superficial cues on explanation quality are more severe when time and prior knowledge are limited [45, 57]. In the absence of effort and expertise, users will inevitably rely on superficial cues to explanation quality, such as fluency [111], a characteristic that LLM explanations typically possess in spades. This suggests a potential tension in providing LLM explanations to lay users: the properties that make such explanations intelligible and compelling may be precisely those that lead to overreliance. As such, we encourage LLM explanations to be evaluated and optimized for appropriate reliance, in addition to other qualities such as fluency, justification quality, and satisfaction.

Sources. Our results offer some basis for optimism, however: sources helped reduce overreliance on incorrect answers and increase appropriate reliance on correct answers. One possibility is that sources encouraged participants to engage in slow and careful System 2 thinking, instead of quick and automatic System 1 thinking [52, 53]. In our study, participants spent significantly more time on task when provided with sources, especially when the LLM’s answer was incorrect. The qualitative data also supports this. Many participants wrote that they checked sources. Many also wrote that they submitted a different answer from the LLM’s answer when it conflicted with the information in the sources. We emphasize, however, that the sources provided in Study 2 were all real and tended to provide accurate and relevant information. This is not always the case. Recent work has found that popular LLM-infused applications frequently generate statements that are not supported by sources [77] and sometimes even generate fake sources [2]. If the provided sources are junk or just broken links, then presumably they will not help foster appropriate reliance. They could potentially even hurt by making the LLM response look more trustworthy, similar to how flawed and meaningless explanations have been found to increase people’s trust and reliance [31, 55, 104]. In addition to improving the quality of sources in LLM responses, future work should explore different issues with sources (e.g., fake, unreliable, conflicting sources and inaccurate summaries of sources), design choices (e.g., location of sources and amount of preview), and their effects on people’s perceptions and behaviors. When it comes to choosing between providing sources only and providing sources and an explanation, there are benefits and drawbacks to each. When the LLM answer is incorrect, participants’ accuracy is highest on responses with sources only (M= 31.8%), followed by responses with explanation and sources (M= 23.1%), neither (M= 21.8%), and explanation only (M= 17.2%) — suggesting that providing sources only is most effective at reducing overreliance on incorrect answers.

Conclusion. We conducted two empirical studies to understand how different features of LLM responses shape users’ reliance. We found that the presence of explanations increases reliance on both correct and incorrect responses. However, we observed less reliance on incorrect responses when sources are provided or when explanations exhibit inconsistencies. Our findings highlight the importance of evaluating LLM response features with users before deployment. Our findings also suggest that providing (accurate and relevant) sources and designing interventions that help users notice and reason about inconsistencies and other unreliability cues in explanations can be promising directions for fostering appropriate reliance on LLMs.

Limitations. 5.3 Limitations There are several limitations of our work that are worth reflecting on. First and foremost, our studies were conducted in the context of objective question-answering and may not generalize to other contexts of LLM use (e.g., writing, idea generation, and task automation). We encourage the community to conduct more empirical studies on how user reliance is shaped in various contexts. Each of our studies has a different set of strengths and limitations. Study 1 was a think-aloud study that offered descriptive examples of how users interpret and act upon different LLM response features in a relatively natural setting. However, prior work has pointed out that the set-up of a think-aloud study can cause people to behave differently than they would otherwise [9, 34, 90]. For example, we saw a much higher rate of source clicking in Study 1 (M = 63.8%) than in Study 2 (M = 25.8%) which was an online experiment. We also emphasize that the LLM response features identified in Study 1 are not comprehensive. We suggest future work to explore what other features influence users’ reliance and can help them succeed in tasks despite inaccuracies from LLMs.