To Trust or to Think: Cognitive Forcing Functions Can Reduce Overreliance on AI in AI-assisted Decision-making

People supported by AI-powered decision support tools frequently overrely on the AI: they accept an AI’s suggestion even when that suggestion is wrong. Adding explanations to the AI decisions does not appear to reduce the overreliance and some studies suggest that it might even increase it. Informed by the dual-process theory of cognition, we posit that people rarely engage analytically with each individual AI recommendation and explanation, and instead develop general heuristics about whether and when to follow the AI suggestions. Building on prior research on medical decision-making, we designed three cognitive forcing interventions to compel people to engage more thoughtfully with the AI-generated explanations. We conducted an experiment (N=199), in which we compared our three cognitive forcing designs to two simple explainable AI approaches and to a no-AI baseline. The results demonstrate that cognitive forcing significantly reduced overreliance compared to the simple explainable AI approaches. However, there was a trade-off: people assigned the least favorable subjective ratings to the designs that reduced the overreliance the most. To audit our work for intervention-generated inequalities, we investigated whether our interventions benefited equally people with different levels of Need for Cognition (i.e., motivation to engage in effortful mental activities). Our results show that, on average, cognitive forcing interventions benefited participants higher in Need for Cognition more. Our research suggests that human cognitive motivation moderates the effectiveness of explainable AI solutions.

Introduction. From loan approval to disease diagnosis, humans are increasingly being assisted by artificially intelligent (AI) systems in decision-making tasks. By combining two types of intelligence, these emerging sociotechnical systems (i.e., human+AI teams) were expected to perform better than either people or AIs alone [35, 36]. Recent studies, however, show that although human+AI teams typically outperform people working alone, their performance is usually inferior to the AI’s [2, 7, 9, 26, 30, 41, 59]. There is evidence that instead of combining their own insights with suggestions generated by the computational models, people frequently overrely on the AI, following its suggestions even when those suggestions are wrong and the person would have made a better choice on their own [9, 30, 41]. Explainable AI (XAI), an approach where AI’s recommendations are accompanied by explanations or rationales, was intended to address the problem of overreliance: By giving people an insight into how the machine arrived at its recommendations, the explanations were supposed to help them identify the situations in which AI’s reasoning was incorrect and the suggestion should be rejected. However, evidence suggests that explainable systems, also, have not had substantial success in reducing human overreliance on the AIs: when the AI suggests incorrect or suboptimal solutions, people still on average make poorer final decisions than they would have without AI’s assistance [2, 7, 30, 40, 67]. We posit that the dual-process theory provides a useful lens through which to understand the reasons why the explanations do not eliminate human overreliance on AIs. According to the dual-process theory, humans mostly operate on System 1 thinking, which employs heuristics and shortcuts when making decisions [32, 63]. Because most of the daily decisions are successfully accomplished by these heuristics, analytical thinking (i.e., System 2) is triggered rarely, as it is slower and costlier in terms of effort. Yet, System 1 thinking leaves us vulnerable to cognitive biases that can result in incorrect or suboptimal decisions [32]. We do not want to suggest, however, that System 1 thinking is always bad or inappropriate. Indeed, successful use of pattern matching and heuristics that arise from extensive experience are important components of expertise [49]. But even experts can fall prey to cognitive biases [42] so a judicious combination of both processes is needed. In the context of explainable AI, the implicit assumption behind the design of most systems is that people will engage analytically with each explanation and will use the content of these explanations to identify which of the AI’s suggestions are plausible and which appear to be based on faulty reasoning. Because evaluating every explanation requires substantial cognitive effort, which humans are averse to [38], this assumption is likely incorrect. Instead, people appear to develop heuristics about the competence of the AI partner overall. Indeed, some studies demonstrate that explanations are interpreted as a general signal of competence—rather than being evaluated individually for their content—and just by their presence can increase the trust in and overreliance on the AI [2]. We argue that to reduce human overreliance on the AI and improve performance, we need to not only develop effective explanation techniques, but also ways to increase people’s cognitive motivation for engaging analytically with the explanations. A number of approaches have been explored in other domains for engaging people in more analytical thinking to reduce the impact of cognitive biases on decision-making. One of the most promising appears to be the cognitive forcing functions—interventions that are applied at the decision-making time to disrupt heuristic reasoning and thus cause the person to engage in analytical thinking [42]. Examples of cognitive forcing functions include checklists, diagnostic time-outs, or asking the person to explicitly rule out an alternative. Bringing these insights into AI-assisted decision-making, we hypothesized that adding cognitive forcing functions to existing explainable AI designs would help reduce human overreliance on the AI. An important aspect of the design of forcing functions in this context, however, is the usability and the acceptability of these interventions. We hypothesized that stricter interventions will push people to think harder, but will be found complex and less usable. We conducted an experiment with 199 participants on Amazon Mechanical Turk, in which we compared three cognitive forcing designs to two simple explainable AI approaches, and to a no-AI baseline. Our results demonstrate that cognitive forcing functions significantly reduced overreliance on the AI compared to the simple explainable AI approaches.

Related work. Dual processing theory postulates that humans make decisions through one of two distinct cognitive processes — fast and automatic (i.e., System 1 thinking) or slow and deliberative (i.e., System 2 thinking) [32, 33, 63]. System 1 thinking employs heuristics that lead to effective decisions in most daily decision-making settings. These heuristics are highly useful and mostly effective, however they can lead to systemic and predictable errors [34]. Importantly, as a definition of their expertise, experts also develop heuristics, which at times make them prone to faulty decisions even in highstakes domains [42]. Therefore, shifting people’s reasoning from System 1 to a more deliberative and rational, System 2 thinking remains a challenge that researchers have tackled in numerous high-stakes domains [14, 15, 42].

For example, substantive effort has been spent in clinical settings to improve diagnostic outcomes by eliciting analytical thinking from clinicians. That is because evidence suggests that very often diagnostic errors are not cases of lack of knowledge, but rather of biases and faulty information synthesis [25]. Interventions that seek to mitigate these types of errors by invoking deliberative thinking can be categorized into educational strategies and cognitive forcing functions [16]. Educational strategies are metacognitive debiasing techniques that aim at enhancing future decision-making by increasing clinicians awareness about the existence of different decisionmaking pitfalls. They are introduced through educational curricula, simulation training, and other instruction techniques. On the other hand, cognitive forcing functions are interventions which take place at the decision-making time and encourage the decision-maker to engage analytically with the decision at hand. For instance, in clinical settings, these forcing functions can take the form of checklists, diagnostic time-outs, and slow decision-making. Studies indicate that cognitive forcing functions, as interventions at the time of decision-making, are more effective than educational strategies in increasing accuracy of diagnostic processes in clinical decision making [42, 55].

2.2 Overreliance in AI-assisted decision-making As machine learning models have achieved high accuracies across numerous domains, they are increasingly being deployed as decision-support aids for humans. The implicit assumption is that because these models have high accuracies, so will the overall human+AI teams.

Method. We conducted an experiment with 3 different cognitive forcing interventions, two simple explainable AI conditions, and a no-AI baseline, to examine whether cognitive forcing functions are successful in reducing human overreliance on the AI when working on a decision-making task. Specifically, we hypothesized that:

H1a: Compared to simple explainable AI approaches, cognitive forcing functions will improve the performance of human+AI teams in situations where the AI’s top prediction is incorrect.

And, consequently:

H1b: Compared to simple explainable AI approaches, cognitive forcing functions will improve the performance of human+AI teams.

However, because cognitive forcing functions cause people to exert extra cognitive effort, we expected that there would be a trade-off between the acceptability of the design of the human+AI collaboration interface and its effectiveness in reducing the overreliance: people would over-rely less on the AI when they were forced to think harder, but they would prefer such interfaces less than those that require less thinking. Thus, we hypothesized that:

3.1 Task description Accessing experts such as judges, or clinicians for experiments is notoriously challenging and costly. We, therefore, designed the task around nutrition as an approachable domain for laypeople.

Participants were shown meal images and were asked to replace the ingredient highest in carbohydrates on the plate, with an ingredient that was low in carbohydrates, but similar in flavor. Each of the meals had an ingredient with substantially more carbohydrates than other ingredients on the plate. Participants had to make two discrete choices: First, they had to pick the ingredient to take out from the meal (by selecting from a list of approximately 10 choices, that included all the main ingredients in the image plus several other ingredients). Next, they had to select a new ingredient to put in place of the removed ingredient (again, by selecting from a list of roughly 10 choices).

3.2 Conditions We designed six conditions that varied in whether and how a simulated AI assisted the participants in making their decisions.

No AI. In the no AI condition, participants were shown just the image of the meal and the two pull-down menus to select which ingredient to remove and which new ingredient to put in its place.

3.2.1 Simple Explainable AI (SXAI) Baselines.

Explanation. The explanation condition (Figure 1a) included the list of the ingredients that the simulated AI (described in the next section) recognized on the plate and top four substitutions for the ingredient highest in carbohydrates among the recognized ingredients. Each substitution was accompanied by a specific feature-based explanation showing the estimated carbohydrate reduction and flavor similarity. While there is a great diversity in the designs of explanations in the explainable AI community, this condition reflects a common approach for designing the human-AI collaborative interface: the explanation and the AI suggestion are shown immediately to the human decision-maker.

Uncertainty. The uncertainty condition was like the explanation, except that participants were also shown a confidence prompt “The AI is X % confident in its suggestion.” (Figure 1b). The prompt was shown only when the AI was uncertain about its suggestion. This condition captures another common approach for designing interfaces for human-AI collaboration [2, 40, 66].

3.2.2 Cognitive Forcing Functions (CFF).

On demand. In the on demand condition, the AI suggestion was not shown to the users by default. Users could see the suggestion and the explanation (identical to the one in the explanation condition) if they clicked on the “See AI’s suggestion” button (Figure 1c). We hypothesized this to be a light form of cognitive forcing function, where the human would engage with the explanation because they explicitly requested it.

Update. Participants in the update condition had to make the initial decision without the help of an AI (i.e., like in the no AI condition). Having made the initial decision, they were shown the AI’s suggestion and explanation and could update their decision.

Discussion. Consistent with prior research [7, 9, 26, 41], our results demonstrate that people aided by simple explainable AI interfaces performed better overall than people who completed the task without any AI support. Also consistent with the prior research, these human+AI teams performed worse than the AI model alone (its accuracy was set to 75%). Our results suggest that part of the reason is that people frequently overrelied on the AI: they followed the suggestions of the AI even when its predictions were incorrect. Consequently, when the AI predictions were incorrect, people aided by the simple explainable AI approaches performed less well than people who had no AI support. Our results demonstrate that cognitive forcing functions reduced overreliance on AI compared to the simple explainable AI approaches: When the simulated AI model was incorrect, participants disregarded the AI suggestions and made the optimal choices significantly more frequently in the cognitive forcing function conditions than in the simple explainable AI conditions. These results support the hypothesis (H1a). However, even with the cognitive forcing functions, the human+AI teams in our study continued to perform worse than the AI model alone: the cognitive forcing functions reduced, but not yet eliminated, overreliance on the AI. Our results did not provide support for the hypothesis (H1b) that cognitive forcing functions will improve the performance of human+AI teams. There was no significant difference in performance between simple explainable AI approaches and cognitive forcing functions. Subjective ratings of the conditions indicate that participants in cognitive forcing function conditions perceived the system as more complex than those in simple explainable AI conditions. The tension between subjective measures and performance is observed in the correlation analyses of the subjective measures versus the performance. When the AI model was making incorrect predictions, people performed best in conditions that they preferred and trusted the least, and that they rated as the most difficult. Thus, as hypothesized (H2), there appears to be a trade-off between the acceptability of a design of the human-AI collaboration interface and the performance of the human+AI team. Overall, our research suggests that human cognitive motivation moderates the effectiveness of explainable AI solutions. Hence, in addition to tuning explanation attributes such as soundness and completeness [39], or faithfulness, sensitivity, and complexity [5], explainable AI researchers should ensure that people will exert effort to attend to those explanations. Our results also lend further support to the recent research that demonstrated that using proxy tasks (where participants are asked, for example, to predict the output of an algorithm based on the explanations it generated) to evaluate explainable AI systems may lead to different results than testing the systems on actual decision-making tasks (where participants are asked to make decisions aided by an algorithm and its explanations) [7]. To explain their results, the authors hypothesized that proxy tasks artificially forced participants to pay attention to the AI, but when participants were presented with actual decision-making tasks, they focused on making the decisions and allocated fewer cognitive resources to analyzing AI-generated suggestions and explanations. Our results showed that cognitive forcing interventions improved participants’ ability to detect AI’s errors in an actual decision-making task, where the human is assisted by the AI while making decisions.

Conclusion. Third, our research suggests that as researchers, we should audit our work for emerging inequalities among relevant groups. Our results show that cognitive forcing functions disproportionately benefited participants with high Need for Cognition — a group that has been shown in other settings to benefit the most from useful but cognitively demanding user interface features. Together, these findings suggest that research in explainable AI should not assume that participants will engage with the explanations by default. Hence, more effort should be spent on devising interventions that elicit analytical thinking and engagement with explanations when necessary to avoid unquestioning trust from humans. Also, further research is necessary to account for individual differences in cognitive motivation and explore the right amount and timing of cognitive forcing necessary for optimal human performance with AI-powered decision-support tools.

Limitations. 5 ETHICAL CONSIDERATIONS: INDIVIDUAL DIFFERENCES IN COGNITIVE MOTIVATION We are mindful of the fact that technological interventions can lead to intervention-generated inequalities (IGIs). IGIs arise when the benefits of an intervention disproportionately accrue to a group that is already privileged in a particular context [60]. Thus, even though everyone might benefit to some extent, the gaps between different groups increase. Therefore, we conducted a limited internal audit [54] of our work to check for inequalities that might emerge and to understand how different groups of people are affected by the introduction of cognitive forcing functions into the AI-assisted decision-making processes.