Why can an AI's explanation make you more confident in a wrong answer than no explanation at all?

This explores why an AI's explanation, offered on its own with nothing to check it against, can leave people judging worse than no transparency feature at all. The collection doesn't hold the specific head-to-head experiment your question describes, but several notes point to the same cause. An explanation feels like evidence without actually being evidence. It makes people more confident, and it doesn't make them any better at telling right answers from wrong ones.

The clearest case comes from chain-of-thought, the step-by-step reasoning that models show before answering. In a 100-person study, the reasoning traces that helped the model most were the ones people found hardest to follow. Those traces also made people *more* likely to accept wrong answers Do chain-of-thought traces actually help users understand model reasoning?. That is how 'explanation only' can lose to 'nothing'. With no explanation, a reader stays somewhat skeptical. With a long, self-correcting explanation, the reader sees signs of careful thought and stops checking. Another finding makes this worse: most of a reasoning trace isn't doing the reasoning. Stripped-down 'draft' chains reach the same accuracy using just 7.6% of the tokens, so the rest is mostly style and narration Can minimal reasoning chains match full explanations?. Much of what readers take as a window into the model's reasoning is closer to presentation.

The contrast is between explanations and verification signals. In a study of how people judge AI-written text, readers with no cues about where claims came from couldn't tell truth from fabrication at all. They fell for fluent hallucinations as easily as for real facts. A display showing which claims had been verified, and how densely, restored their ability to tell the difference Can readers tell truth from fabrication without evidence signals?. The design lesson: features that show *what can be checked* help people judge, while features that show *how the model got there* mostly change how the answer comes across. Retrieval systems reach a similar conclusion from the engineering side. The most robust ones refuse to answer when there's no evidence behind a claim, instead of producing a confident account anyway Can RAG systems refuse to answer without reliable evidence?.

Two more notes reframe the question. One argues that explainable AI is a communication problem, not a transparency problem. Whether an explanation helps depends on who presents it, how it's framed, and what the reader is trying to do, not only on what it says What if XAI is fundamentally a communication problem?. An explanation shown without that context can easily mislead. The other, from philosophy of science, argues that model opacity only matters when you treat the output as a justified claim. In some scientific work, opaque predictions led to discoveries that were then checked independently Can opaque models guide discovery without needing interpretation?. Taken together, the surprising takeaway is that the goal isn't to make AI explain itself. It's to make AI outputs checkable. A bare explanation can make people trust an answer more without giving them anything to check it with.