AI explanations are supposed to earn your trust — why do they often make people trust bad answers even more?

This explores why showing people how an AI reached its answer often doesn't stop them from over-trusting it, and what actually helps. The surprising headline from the corpus: explanations don't just fail to reduce automation bias. They often make it worse. A review of 35 studies found that explainable-AI features frequently deepen misplaced trust, especially among users with low AI literacy (Does explaining AI decisions actually reduce user overreliance on it?). An explanation can look like proof of competence, so it ends up reassuring people instead of prompting them to check.

Part of the reason is that over-reliance isn't one bias. Several pile on top of each other. The AI's first suggestion becomes an anchor that later judgment drifts toward. Meanwhile, people who know the least about a topic are the worst at recognizing when to doubt the AI, which is the Dunning-Kruger pattern (How do anchoring and Dunning-Kruger deepen AI overreliance?). An explanation does nothing about either problem. If you can't evaluate the reasoning, a well-written rationale just gives you more reasons to agree. The anchoring effect can work below awareness, too: biased AI autocomplete suggestions shifted people's attitudes more than the same arguments shown as plain text, and participants didn't notice it happening (Do AI autocomplete suggestions shift attitudes more than static text?).

A second idea reframes the problem. Maybe explanation is a communication act and not a transparency feature. How well an explanation works depends on who presents it, how it's framed, and what role the listener is in. The quality isn't contained in the explanation itself (What if XAI is fundamentally a communication problem?). Related work warns that the most dangerous systems are the ones that seem competent: fluent output wears down skepticism quietly (How do competent systems quietly undermine safety oversight?), and more automation tends to hide errors behind polished results without removing them (Does more automation actually hide rather than eliminate errors?). A tidy explanation can be one more layer of that polish.

What seems to work instead is changing the person's role, not the AI's output. The XAI review found that active engagement and deliberate verification reduce over-reliance where transparency alone doesn't (Does explaining AI decisions actually reduce user overreliance on it?). One design, Learning to Guide, has the AI point out which parts of the input matter and leaves the decision to the human. This removes the anchor entirely (Can AI guidance reduce anchoring bias better than AI decisions?). Older automation research points the same way: a meta-analysis found that harm concentrates when machines move from analyzing information to choosing actions, because people who only monitor lose the awareness they need to step in when something fails (When does automation harm performance after failures?). Feedback also beats disclosure. Telling users they're working with an AI changed nothing lasting on its own, while repeatedly seeing actual outcomes did recalibrate their trust (Does revealing AI identity help or hurt user trust?).

The less obvious takeaway is that explanations can serve the system more than the user. When an automated system fails, blame tends to land on the human who was nearby, even if that person had little real control. Researchers call this a 'moral crumple zone' (Do automated systems shift blame onto their human operators?). An explanation the operator 'saw' makes that blame easier to assign, without making the operator any better at catching the error. If the goal is better human judgment, the corpus suggests designing for active involvement and visible outcomes, not more detailed rationales.