Why doesn't showing people an AI's reasoning stop them from trusting it too much?

This explores why people keep deferring to AI even when they could read an explanation of how it reached its answer. The surprising answer from the corpus is that explanations often make the problem worse. A review of 35 studies found that explainable-AI features frequently *deepen* misplaced trust, especially among people with low AI literacy. An explanation can work as a reassurance signal rather than an invitation to check. What actually reduces overreliance is active engagement: making users verify or reason through the case themselves Does explaining AI decisions actually reduce user overreliance on it?.

Part of the reason is that automation bias doesn't act alone. The same review describes it compounding with anchoring, where the AI's first suggestion becomes the reference point every later judgment is measured against. It also compounds with the Dunning-Kruger effect: the people who most need to doubt the AI are the ones least able to tell when doubt is warranted How do anchoring and Dunning-Kruger deepen AI overreliance?. An explanation can't fix either one. If you've already anchored, you read the explanation as confirmation. If you lack the knowledge to evaluate it, a plausible-sounding rationale looks the same as a correct one. Presentation matters too. Fluent prose, confident tone and clean formatting create a halo effect, and chat interfaces nudge you toward the next question instead of back to the sources Does confident AI tone make users skip fact-checking?.

A useful reframe: an explanation is a message, not a property of the model. Whether it helps depends on who presents it, how it's framed, and what role the reader thinks they're playing What if XAI is fundamentally a communication problem?. A reader positioned as someone approving a decision will skim it. A reader positioned as the person responsible for the decision will interrogate it. This also explains why influence can work without anyone noticing. Biased autocomplete suggestions shifted people's attitudes more than the same arguments shown as plain text, and participants didn't realize it had happened Do AI autocomplete suggestions shift attitudes more than static text?. When the AI is woven into what you're doing, there's no clear moment where you stop and evaluate it.

The structure of the task matters too. A meta-analysis of automation research found the biggest harms come when automation shifts from analyzing information to choosing actions. People who only monitor an automated decision-maker lose the awareness and skill they'd need to step in When does automation harm performance after failures?. One proposed fix flips the arrangement: instead of the AI deciding and the human approving, the AI points out which parts of the input deserve attention and the human decides. This approach, called Learning to Guide, is designed to remove the anchor altogether Can AI guidance reduce anchoring bias better than AI decisions?.

The twist you may not have expected is that the human often ends up taking the blame anyway. When automated systems fail, responsibility tends to land on the nearest operator, even when they had little real control or information. Elish calls this a "moral crumple zone" Do automated systems shift blame onto their human operators?. So an available explanation can end up mostly as cover. It lets everyone say the human could have checked, while the system's design made checking unlikely.