How do anchoring and Dunning-Kruger deepen AI overreliance?
Does automation bias persist not just from lack of information, but through active cognitive mechanisms like anchoring effects and self-overestimation? Understanding these compounding biases could reveal why transparency alone fails to reduce overreliance.
Giuseppe Romeo and Daniela Conti's systematic review of 35 studies on automation bias (AB) — overreliance on automated recommendations — reports that AB "frequently interacts with other cognitive biases, particularly the anchoring effect, where initial AI recommendations disproportionately shape subsequent decisions." The review states this "leads decision-makers to seek evidence that confirms the AI initial suggestion," citing Gaube et al. (2021) and Nourani et al. (2022), "especially when incorrect AI guidance is provided before they form their own judgment," citing Agudo et al. (2024). Separately, the review links AB "to the Dunning–Kruger effect, where individuals with limited AI knowledge overestimate their understanding of it, which makes them more prone to overreliance on automation," citing Horowitz and Kahn (2024).
In the review's account, these are distinct mechanisms that both push in the direction of more reliance, not less. Anchoring operates through the order and framing of advice: when an AI's suggestion arrives before a person has formed an independent judgment, that suggestion becomes the reference point subsequent reasoning confirms rather than tests. The review notes the same anchoring logic can flip into "automation complacency" when the AI's recommendation, offered as a second opinion, happens to align with what the person already believed — the bias looks different depending on sequence, but the result is the same reduced scrutiny. Dunning-Kruger operates through self-assessment: people who know the least about how an AI system works are the ones least able to judge when to doubt it, which the review distinguishes from AI literacy as a protective factor — literacy reduces AB directly, while the Dunning-Kruger pattern describes why low-literacy users do not recognize their own exposure to it.
This sits alongside Does explaining AI decisions actually reduce user overreliance on it?, the sibling note from this review: that note covers the review's claim that explanations and transparency often fail to fix AB, while this note covers the review's account of why AB persists even when people are not simply trusting a system blindly — anchoring and self-overestimation are active cognitive mechanisms, not mere absence of information. It also connects to Do users worldwide trust confident AI outputs even when wrong?: that study's finding that confidence signals dominate accuracy tracking is one plausible channel through which anchoring operates, since a confidently stated first answer is exactly the kind of initial recommendation the review describes becoming the anchor. And it bears on Does unmodified chatbot behavior block rule discovery through sampling bias?, where confirmatory sampling toward an initial hypothesis parallels the confirmatory evidence-seeking the review attributes to anchoring.
The review itself flags a limitation worth carrying forward: a subset of the studies it covers (Alon-Barkat and Busuioc 2023; Selten et al. 2023; Bashkirova and Krpan 2024) argues that what looks like AB in experts may instead be confirmation bias — experts accepting AI advice that matches their prior judgment and resisting advice that does not — which would locate the mechanism in the expert's own prior belief rather than in anchoring to the AI's suggestion per se. The review responds only that those three studies "fail to account for cases where participants rely on incorrect AI recommendations despite their expressed reluctance." Both the anchoring and Dunning-Kruger claims here are drawn from citations within the review rather than from experiments the review ran itself, so the strength of each claim rests on the cited primary studies, not on independent replication by Romeo and Conti.
Inquiring lines that read this note 5
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
Does polished AI output mislead users about actual accuracy and expertise? How do humans calibrate and maintain trust in AI systems?Related concepts in this collection 5
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
-
Does explaining AI decisions actually reduce user overreliance on it?
Explainable AI is designed to build appropriate trust, but does transparency about how AI reaches recommendations actually help users catch mistakes and avoid automation bias, or does it backfire?
sibling note from the same review, covering its claim about explanation design rather than bias mechanism
-
Do users worldwide trust confident AI outputs even when wrong?
Explores whether the tendency to over-rely on confident language model outputs transcends language and culture. Understanding this pattern is critical for designing safer human-AI interaction across diverse linguistic contexts.
confidence-driven reliance is a plausible channel for the anchoring effect described here
-
Does unmodified chatbot behavior block rule discovery through sampling bias?
Do chatbots suppress discovery of hidden rules by sampling examples that confirm user hypotheses rather than challenge them? This matters because it suggests AI overconfidence is manufactured by what evidence surfaces, not just how confidently it's phrased.
parallel confirmatory-seeking mechanism in a different experimental setting
-
How much does wrong AI advice harm radiologist accuracy?
When mammography radiologists receive incorrect AI suggestions labeled as system output, how much does their diagnostic accuracy decline? This matters for understanding automation bias in clinical workflows.
Evidence for: mislabeled AI suggestions cut experienced radiologists' accuracy from 82% to 45.5%, showing anchoring-driven automation bias
-
Do forcing functions reduce AI overreliance without harming team performance?
This study tests whether cognitive forcing functions—interventions that disrupt quick reliance on AI suggestions—can reduce overreliance while maintaining or improving human-AI team performance on decision tasks.
Evidence for: cognitive forcing functions curb overreliance from these compounding biases, though team performance still trails AI alone and users dislike them
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- Exploring automation bias in human-AI collaboration: a review and implications for explainable AI
- The Decision to Verify: How Warmth and User Characteristics Shape Reliance on Conversational Agents for Information Search
- How AI Impacts Skill Formation
- The effects of over-reliance on AI dialogue systems on students' cognitive abilities: a systematic review
- To Trust or to Think: Cognitive Forcing Functions Can Reduce Overreliance on AI in AI-assisted Decision-making
- Automation Bias in AI-Assisted Medical Decision-Making under Time Pressure in Computational Pathology
- Biased AI writing assistants shift users' attitudes on societal issues
- AI Assistance Reduces Persistence and Hurts Independent Performance
Original note title
automation bias compounds with anchoring and the Dunning-Kruger effect to deepen AI overreliance