Exploring automation bias in human-AI collaboration: a review and implications for explainable AI

Source: Giuseppe Romeo and Daniela Conti; AI & Society (open access) · 2025-07-03

As Artificial Intelligence (AI) becomes increasingly embedded in high-stakes domains such as healthcare, law, and public administration, automation bias (AB)—the tendency to over-rely on automated recommendations—has emerged as a critical challenge in human–AI collaboration. While previous reviews have examined AB in traditional computer-assisted decision-making, research on its implications in modern AI-driven work environments remains limited. To address this gap, this research systematically investigates how AB manifests in these settings and the cognitive mechanisms that influence it. Following PRISMA 2020 guidelines, we reviewed 35 peer-reviewed studies from SCOPUS, ScienceDirect, PubMed, and Google Scholar. The included literature, published between January 2015 and April 2025, spans fields such as cognitive psychology, human factors engineering, human–computer interaction, and neuroscience, providing an interdisciplinary foundation for our analysis. Traditional perspectives attribute AB to over-trust in automation or attentional constraints, resulting in users perceiving AI-generated outputs as reliable. However, our review presents a more nuanced view. While confirming some prior findings, it also sheds light on additional interacting factors such as, AI literacy, level of professional expertise, cognitive profile, developmental trust dynamics, task verification demands, and explanation complexity. Notably, although Explainable AI (XAI) and transparency mechanisms are designed to mitigate AB, overly technical, cognitively demanding, or even simplistic explanations may inadvertently reinforce misplaced trust, especially among less experienced professionals with low AI literacy. Taken together, these findings suggest that although explanations may increase perceived system acceptability, they are often insufficient to improve decision accuracy or mitigate AB. Instead, user engagement emerges as the most feasible and impactful point of intervention. As increased verification effort has been shown to reduce complacency toward AI mis-recommendations, we propose explanation design strategies that actively promote critical engagement and independent verification. These conclusions offer both theoretical and practical contributions to bias-aware AI development, underscoring that explanation usability is best supported by features such as understandability and adaptiveness.

Across studies, trust and task complexity are confirmed as predominant factors in AB. Initial trust in AI, coupled with perception of systems reliability, significantly influences reliance on automation (Buçinca et al. 2021; Duan et al. 2024; Horowitz and Kahn 2024). Several factors shape this perceived reliability, including perceived benefit (Kücking et al. 2024), usefulness (Jacobs et al. 2021; Cao et al. 2023), competence (Naiseh et al. 2023), transparency (Vered et al. 2023), absence of uncertainty (Gaube et al. 2021), and the apparent objectivity of AI recommendations (Keding & Meissner 2021). Overreliance is further reinforced by high workload (Vered et al. 2023; Zhang et al. 2024; Alami et al. 2025), time pressure (Cao et al. 2023), and task complexity (Bond et al. 2018; Kupfer et al. 2023; Cecil et al. 2024). Collectively, these elements contribute to situational trust, which correlates with advice conformity (Klingbeil et al. 2024, Küper et al. 2025).

Another mediator factor is the sequence in which advice is given. Positive first impressions can foster excessive trust in automation (Nourani et al. 2022; Vered et al. 2023). The timing of explanations also matters: AI-first protocols may increase user compliance with AI recommendations (Cabitza et al. 2023; Agudo et al. 2024). Alternatively, human-first protocols can lead to automation complacency when the AI suggestions align with the human’s initial evaluation (Cabitza et al. 2023).

At the same time, several moderator factors have been identified that mitigate the likelihood of AB: professional experience (Rezazade Mehrizi et al. 2023; Kücking et al. 2024; Kim et al. 2025) and domain expertise (Nourani et al. 2022; Dikmen and Burns 2022) are the most protective. Furthermore, advanced AI background knowledge exhibits more calibrated trust (Horowitz and Kahn 2024). Finally, verification-related cognitive engagement serves as a critical debiasing mechanism (Kupfer et al. 2023).

AB frequently interacts with other cognitive biases, particularly the anchoring effect, where initial AI recommendations disproportionately shape subsequent decisions. This bias leads decision-makers to seek evidence that confirms the AI initial suggestion (Gaube et al. 2021; Nourani et al. 2022), especially when incorrect AI guidance is provided before they form their own judgment (Agudo et al. 2024). Otherwise, it can also foster automation complacency when the AI’s recommendation, provided as a second opinion, aligns with users’ initial judgment (Cabitza et al. 2023).

AB has also been linked to the Dunning–Kruger effect, where individuals with limited AI knowledge overestimate their understanding of it, which makes them more prone to overreliance on automation (Horowitz and Kahn 2024).

Transparency mechanisms, such as uncertainty estimates, have been proposed to counteract overreliance by highlighting system limitations (Bond et al. 2018; Gaube et al. 2021; Dratsch et al. 2023). However, current methods implemented through XAI have often shown that explanations themselves can be used as heuristics (Rezazade Mehrizi et al. 2023) and reinforce trust in AI systems, exacerbating AB when the advice provided is incorrect (Cabitza et al. 2023; Vered et al. 2023; Cecil et al. 2024). Multiple explanation types (Naiseh et al. 2023) and trust calibration feedback (Tatasciore and Loft 2025) are proven not to help users recognize incorrect AI recommendations as well.

Developing a better understanding of AI capabilities and limitations has been shown to foster better decision-making and reduce AB (Kupfer et al. 2023). Higher levels of AI literacy enable appropriate reliance on automation (Horowitz and Kahn 2024).

Individual differences significantly influence the manifestation of AB. Professional experience and domain expertise play a crucial role, with experts generally less prone to AB, although they can still be affected under conditions of high cognitive load (Bond et al. 2018; Nourani et al. 2022; Dratsch et al. 2023). Personality traits, particularly need for cognition, have been shown to enhance critical engagement and reduce susceptibility to AB (Buçinca et al. 2021). Additionally, familiarity with AI, conceptions about how it works, and prior exposure to it are key factors influencing how individuals interact with and rely on AI systems (Nourani et al. 2022; Horowitz and Kahn 2024).

Demographic factors, such as gender, have also been found to influence AB, with female participants exhibiting lower susceptibility, whereas age appears to have a neutral effect (Kücking et al. 2024).

Our research has also led to a subset of studies (Alon-Barkat and Busuioc 2023; Selten et al. 2023; Bashkirova and Krpan 2024) that challenge the prevailing belief of AB as a distinct phenomenon in human–AI interaction. They suggest that expert participants are more likely to accept AI recommendations that align with their prior judgments while remaining skeptical of those that do not. In summary, these studies argue that confirmation bias primarily drives experts’ acceptance or reluctance of AI recommendations.

In response to these findings, it is worth noting that also Cabitza et al. (2023) observed that in human-first protocols either conservatism or automation complacency (both due to confirmatory heuristics) can occur, depending on whether AI recommendations align with users’ initial assessments. Furthermore, the three studies in question fail to account for cases where participants rely on incorrect AI recommendations despite their expressed reluctance, as highlighted by Gaube et al. (2021). Additionally, these results might be better understood in the context of the developmental trust calibration trajectory proposed by Horowitz and Kahn (2024) and Duan et al. (2024). For instance, Bashkirova and Krpan (2024) themselves acknowledge that mental health practitioners, characteristically, tend to exhibit a greater aversion to adopting AI technologies in their profession.

While XAI and transparency mechanisms are widely proposed as solutions to improve human–AI interaction, the empirical findings thus far presented suggest that current approaches may unintentionally amplify AB by fostering misplaced trust. Key challenges lie in AI literacy, whose lack can limit user’s ability to effectively collaborate with systems (Long and Magerko 2020), and explanation design itself. Merely making AI explainable or transparent is not enough: explanations must actually help users recognize uncertainty or errors (Zhang et al. 2020).

Building on the findings from the reviewed studies, this section explores practical implications for a user interface design that supports human professionals in human–AI collaboration. These proposals seek to solve the three main challenges in hybrid decision-making (cf. 1.1.1; Steyvers and Kumar 2024).

Explanations appear to have limited impact on overreliance when either the task is difficult and the explanation is complex, or when the task is easy and the explanation is simple (Vasconcelos et al. 2023). For high-stakes contexts, Kupfer et al. (2023) highlight the need for explanations that strike a balance between parsimony and detail. Oversimplification may reinforce AB by masking uncertainty and complexity, while overly complex explanations increase cognitive load, reducing users’ ability to evaluate AI outputs effectively.

In this regard, Vered et al. (2023) examined the effects of two types of explanations, local vs. global, on AI-assisted decision-making. Local explanations clarify why an AI model made a specific recommendation, whereas global explanations provide a broader understanding of the inner workings of a system. Their findings suggest that an effective XAI paradigm must balance acceptability (building trust) with skepticism (encouraging critical evaluation). Notably, participants exhibited stronger AB when exposed to local explanations following positive first impressions. However, local explanations also improved focus and attention, illustrating the nuanced and apparently contradictory role of explanation design. Extending this work, Naiseh et al. (2023) compared four explanation types: local, example-based, counterfactual, and global. They found that counterfactual and example-based explanations were especially effective in fostering cognition-based trust, which in turn led to greater engagement with the XAI interface. These findings highlight the critical importance of understandability in explanation design.