To Rely or Not to Rely? Evaluating Interventions for Appropriate Reliance on Large Language Models

Abstract As Large Language Models become integral to decision-making, optimism about their power is tempered with concern over their errors. Users may over-rely on LLM advice that is confidently stated but wrong, or under-rely due to mistrust. Reliance interventions have been developed to help users of LLMs, but they lack rigorous evaluation for appropriate reliance. We benchmark the performance of three relevant interventions by conducting a randomized online experiment with 400 participants attempting two challenging tasks: LSAT logical reasoning and image-based numerical estimation. For each question, participants first answered independently, then received LLM advice modified by one of three reliance interventions and answered the question again. Our findings indicate that while interventions reduce over-reliance, they generally fail to improve appropriate reliance. Furthermore, people became more confident after making incorrect reliance decisions in certain contexts, demonstrating poor calibration. Based on our findings, we discuss implications for designing effective reliance interventions in human-LLM collaboration.

Introduction. As people increasingly incorporate Large Language Models (LLMs) into their decision-making workflows, there is cause for both tremendous optimism and skeptical concern [23, 92]. LLMs encode remarkably vast amounts of knowledge; can communicate with users in clear, natural language; and have the potential to improve both the quality and efficiency of users’ decision-making [75]. However, people can over-rely on LLM advice [33, 71, 73] or, in contrast, under-rely on it if they mistrust their LLM partner [89]. LLMs are prone to hallucination and error, yet often present their outputs with impressive detail and confidence [13, 90]. In contrast, algorithm aversion—the tendency to distrust algorithmic agents—can lead users to ignore helpful recommendations due to general distrust towards the technology [10, 21]. Conscious effort must therefore be placed on helping users calibrate appropriate levels of trust and reliance on LLM advice [91, 93]. Neither over-reliance nor under-reliance is ideal; instead, the focus should be to achieve appropriate reliance [30, 57, 66]. Research in the tangential field of human-AI collaboration has explored methods like explanations and calibration techniques to improve joint decision-making by enhancing AI transparency and verifiability, though results often show mixed outcomes and trade-offs [4, 48, 58, 68, 78]. In this work, we adapt rigorous measures of appropriate reliance from prior works in human-AI collaboration [67] to benchmark the impact of several LLM reliance interventions in human-LLM collaboration.

Due to the risk of hallucinations, reliance and calibration interventions in LLMs are often motivated by curbing over-relying on wrongful information [39, 71, 73], however recent attention has also shifted to optimizing for appropriate reliance [19, 22, 89, 93]. Reliance interventions that involve communicating the capabilities and confidence of an LLM to its users [73], or introducing various types of frictions that the user must bypass to receive the advice [19], are being actively developed and evaluated. Since interventions are evolving rapidly, many of these methods still lack comprehensive evaluations of appropriate reliance in real users, especially those focusing on technical contributions [5, 8, 51]. Furthermore, people’s subjective perceptions—such as their self-confidence—can have a complex relationship with their reliance behaviour [12, 39, 54, 71], which warrants further exploration. Finally, the impact of different intervention types on reliance and perceptions remains unclear, as existing studies do not benchmark multiple interventions. We seek to address these gaps by asking the following research questions:

• RQ1: How do different reliance interventions affect users’ tendencies to over- and under-rely on LLM advice? • RQ2: Can reliance interventions appropriately calibrate users’ confidence in LLM advice?

Summary of Contributions. We conduct a pre-registered, randomized online experiment with n= 400 participants where they attempted two diverse and challenging question-answering tasks: LSAT logical reasoning and image-based numerical estimation. Participants first attempted the tasks on their own, then were shown LLM advice that was randomly treated with one of four conditions (three reliance interventions and one baseline) and given the question again. To evaluate reliance, we implemented a rigorous set of evaluation metrics that isolate the decision-making process users undergo when choosing to rely on the LLM’s advice or their own judgment. We find that while interventions can significantly reduce overreliance, they are generally not effective at improving appropriate reliance. These interventions tend to reduce reliance overall—reducing over-reliance at the expense of useful reliance. Participants were also generally miscalibrated in their confidence, reporting a higher increase in confidence when they make the wrong reliance decision. Out of all the interventions, Reliance Disclaimer is the only one to improve appropriate reliance and better calibrate confidence when participants relied on LLM advice, but this effect was only observed on the LSAT task. Uncertainty Highlighting worsened subjective perceptions and had the least effect on reliance out of the three interventions. Implicit Answer induced better self-reliance in both tasks, but at the cost of requiring significantly more time for participants to process the information. We also discuss the implications of our results for the design and evaluation of appropriate reliance interventions for the future of human-LLM interactions. We highlight the need to perform rigorous, human-centered evaluations of technical LLM calibration methods, particularly with emphasis on achieving appropriate reliance.

Related work. The need for appropriate reliance has been a constant over the years of technological development [46]. With the advancement of AI technologies, much attention has turned on how end users can incorporate assistive AI models into their decision-making tasks [14, 40, 45]. Given that AI predictions are not guaranteed to be accurate, the user must decide whether to or not to incorporate the AI’s advice [12, 17, 32, 54, 61, 72]. The concept of appropriate reliance can typically be defined as "relying on the AI when it’s correct, and relying on yourself when it’s not"—which concerns reducing both overreliance and under-reliance [67, 68]. Root causes for inappropriate reliance can come from misreliance (inherently nonoptimal rate of reliance) or discrimination errors (being unable to identify the payoff of following the AI relative to following oneself) [29, 57]. Factors that influence reliance include the expertise of the user in the task, their cognitive engagement in the task, and the information presented by the AI [14, 28, 30]. On the individual level, people may also be averse or appreciative towards algorithms, perhaps through pre-conceptions or judgments formed after seeing the AI’s performance [21, 37, 53, 57, 62]. A number of strategies have been developed to augment users’ engagement and to provide them with more information of the AI decision, such as through providing explanations [64, 68, 78, 91]. The best interventions for appropriate reliance should be designed to support the users’ abilities to verify the AI’s recommendation, not necessarily to provide a fully transparent system [26, 63, 82]. We adapt appropriate reliance metrics [67] to assess the ability of LLM interventions to help users verify outputs and increase appropriate reliance.

Recent research has investigated the role of LLMs in human-LLM collaborations for applications such as writing [47], communication [27], coding [35, 60], education [7, 43, 52], fact-checking [38, 71, 84], information search [65, 69, 73], and other specialized domains [15, 18, 87]. In these tasks, the LLM can provide guidance and information for the user to assist in their decision-making, learning, and creations [89]. They can also offer antagonistic advice to induce the user into considering alternative options [11, 16, 56]. Even when the user is just reading without explicitly searching for advice, the opinions embedded in the LLM outputs can be influence the user’s In addition to technical limitations of LLMs, human decisionmakers are also imperfect and subject to cognitive biases [6]. People may have unrealistic expectations of LLMs and use them inappropriately [76], such as by assuming LLMs are skilled at math simply because they are computational models [93].

Method. We designed a large-scale, randomized, and incentivized experiment to evaluate the efficacy of three reliance interventions applied to the outputs of GPT-4o1 in two question-answering tasks. To answer our research questions, we will compare the effect of the interventions on performance, reliance behaviour, confidence calibration, and time of engagement against the baseline LLM advice in the Control condition. In this section, we outline our choices of tasks, procedure, measures, and conditions.

We gave the participants two separate question-answering tasks, which we carefully selected to meet the following desiderata:

D1 Accessible to a general audience. The tasks should not require any specialized knowledge or skills, so that they are relevant and accessible to a typical person. D2 Difficult. The tasks should be sufficiently challenging, requiring a meaningful level of cognitive effort to answer correctly. D3 Resistant to cheating. The tasks should not be straightforward to solve by using external assistance, such as a search engine. Driven by the desiderata above, one of our central objectives was to select tasks that a typical person could reason about, even if they do not arrive at the correct answer. We therefore intentionally excluded factual questions characterized by domain-specific knowledge that individuals either possess or lack, and cannot conclude through reasoning alone. This approach emphasizes critical thinking over recall of factual information, and ensures that the average person has the potential to get the correct answer, as well as the ability to engage with LLM advice and reliance interventions. To increase generalizability, we selected two different question formats: multiple-choice, a common standard in human-AI collaboration research; and numerical estimation, which allows for a more granular measurement of the LLM’s influence on participants through the weight-of-advice (WoA). Below, we outline the tasks selected for each format.

LSAT Logical Reasoning. Based on preliminary results from piloting multiple-choice questions from several domains, we found that logical reasoning questions best balanced difficulty with accessibility. We selected LSAT logical reasoning questions as the domain for the multiple-choice task and adapted two multiple-choice LSAT logical reasoning questions, both labeled ‘difficult’, for the first task.

Numerical Estimation. While most previous studies in human-AI collaboration focus on multiple-choice or binary outcome questions, we believe that introducing a numerical estimation task adds a valuable new dimension to this research space. Among the various forms of numerical estimation, such as predicting values in regression tasks, we opted for an image-based format based on a classic carnival game: estimating the quantity of beans in a glass jar. We posit that this is a gamified task that is familiar to most people and does not require advanced mathematics (satisfying D1). Despite the simplicity of the estimation process, it is still challenging to achieve accurate results (satisfying D2). Lastly, since we took the pictures and many of the beans are occluded, it is impossible to use external assistance to “cheat” on this task (satisfying D3). For the experiment, we captured photos of glass jars with manuallycounted quantities of beans from both side perspectives and bottom perspectives. One set featured a smaller jar containing 403 chickpeas, while the other contained a larger jar with 856 jellybeans. The pictures and sample LLM advice can be found in Figures A.3-A.4 in Appendix A. Unlike in multiple-choice tasks, where advice can be objectively classified as good or bad, the goodness of the advice in the numerical estimation setting depends on the participant’s initial estimate. For example, while advice recommending a number that is far lower than the ground truth is objectively wrong, it is still better than an even lower participant estimate. To account for this variability, we provided five distinct levels of LLM advice relative to the ground truth quantity: very low, low, good, high, and very high.

Discussion. We now present the key findings to our research questions on appropriate reliance and confidence calibration. We then extend our findings to broader implications for design and for evaluating reliance on LLMs, and discuss the limitations of our experiments.

Cognitive Forcing Designs Reduce Over-Reliance, but May Not Improve Appropriate Reliance. In our study, we investigated the effectiveness of three interventions on LLM reliance in two diverse and challenging tasks. We use measures of inappropriate reliance, separating over-relying on bad advice from under-relying on helpful advice, to uncover mechanisms of sub-optimal human-LLM team performance. Our findings suggest that cognitive forcing designs in the form of confidence calibration (highlighting uncertain tokens) and usability friction (adding disclaimers and removing direct answers) can help people shift away instinctively relying on the LLM, but improving appropriate advice usage is a harder challenge.

While the effect of each intervention varied by the task, all three achieved a reduction in over-reliance in at least some experiment configurations. However, the only intervention to reduce both overreliance and under-reliance is Reliance Disclaimer in the LSAT logical reasoning task. The Numerical Estimation task captured reliance with more granularity, which provided insight on how the interventions induced under-reliance differently. Our results The increase in advice under-reliance supports the cost-benefit framework from explainable AI literature, which shows that users will not engage with explanations if they are more cognitively complex than the task itself [78]. This suggests that for the LSAT task, static Reliance Disclaimers may have added just enough friction to dissuade habitual over-reliance. However, advice with other interventions, like Implicit Answer in Numerical Estimation, may have been perceived as requiring more effort to use than it was worth. Future work should aim to quantify the comprehension difficulty of the interventions and the tasks to understand how they relate to reliance behaviours. In implementing interventions, designers should consider the targeted use case and develop cognitive forcing techniques that do not increase burden substantially.

Confidence When Using LLMs is Poorly Calibrated. We examine how participants’ confidence in their answers change after seeing the LLM’s advice. Ideally, even with incorrect reliance behaviour, confidence should be lower when individuals are internally uncertain about their decision, such as when they are skeptical of the LLM’s advice. However, we largely observe the opposite or inconclusive effects, where wrong reliance results in higher gains in confidence. This raises particularly problematic implications, suggesting that individuals were overall uncalibrated when receiving assistance from LLMs, corroborating prior findings [71, 89] There were a few areas where the interventions had success. In the Numerical Estimation task, Implicit Answer reduced confidence during under-reliance, while in the LSAT task, Reliance Disclaimer lowered confidence gains during over-reliance, which can be exploited as feedback mechanisms for users to learn from. Both methods present different pathways using which confidence can be influenced by modifying LLM output, but further work is needed to achieve generalizable findings. As shown in prior works in human- AI collaboration, self-confidence and advice agreement rates have nuanced relationships with reliance on advice [54], which can be further investigated in the context of human-LLM collaboration.

Conclusion. We measured the effects of three different LLM interventions on users’ reliance behaviours and confidence levels in two difficult LLM-assisted question-answering tasks. We find that adding a persistent disclaimer is the most effective at improving appropriate reliance (in the LSAT task) without significantly increasing the time needed to engage with the LLM advice, while both highlighting uncertain tokens and removing direct answers only reduced over-reliance and increased the engagement time. In addition, people’s confidence levels were poorly calibrated, generally exhibiting higher confidence increases when they made the wrong reliance decision. Based on our results, we call for the following: rigorous and human-centered evaluation of novel LLM reliance interventions; consideration for trade-offs between over- and under-reliance based on the context of use; and future research in naturalistic human- LLM collaboration tasks.

Limitations. Overall, our results would be further strengthened by future research that explores LLM reliance in more naturalistic and longterm settings, offering a clearer picture of how users assess and use model outputs in the real world. Incorporating qualitative measures can also provided more nuanced feedback on how the participants regarded the advice and made choices.

Engagement of Participants in Tasks. The generalizability of our results is primarily limited by our design of a lab experiment with crowdworkers recruited from Prolific. Since participants knew that they were participating in an experiment, they may have inferred that they were being evaluated on LLM reliance and become more cautious about their reliance behaviours. Although we tried to emulate the risks and benefits of real-world decision-making by rewarding participants for correctness, this may have encouraged participants to “hedge"; to maximize their chances of earning some bonus payment by choosing different answers at each stage.