Why do people trust a smooth, confident-sounding explanation instead of actually checking if the reasoning is right?

This looks at why people treat an AI's explanation as evidence that the AI is right — because it's long, fluent, or well-structured — instead of checking whether the reasoning actually holds up. The research here gives a fairly blunt answer. Most explanations work as confidence boosters, not as tools for checking the answer, and people mostly respond to the surface features. When researchers made LLM explanations longer, readers grew more confident but got no better than chance at telling right answers from wrong ones Does longer explanation text actually improve answer evaluation?. More broadly, the confidence readers take from an explanation's wording barely tracks how confident the model actually is internally, which leaves them steadily overconfident Why do readers trust LLM explanations more than they should?.

Part of the reason is that a typical explanation only argues for one side. Reasoning traces and after-the-fact justifications make people accept AI answers whether or not those answers are correct. The one format that actually helped people catch errors laid out arguments both for and against the answer Do explanations actually help users spot AI mistakes?. Read that way, a one-sided explanation is closer to persuasion than to evidence. Close reading doesn't help much when the text only shows you reasons to agree. A related study found that people don't prefer the formats that would help them verify: they liked planning-and-decomposition reasoning traces best, yet simpler chain-of-thought traces were better for spotting mistakes Do people prefer the reasoning formats that help them verify?. The format that looks most thorough produced more false alarms and more unwarranted trust.

Fluency itself is a strong cue, and it can be faked. Models trained to imitate ChatGPT fooled human evaluators by copying its confident, polished style, without getting any better at facts Can imitating ChatGPT fool evaluators into thinking models improved?. In another study, readers with no source information couldn't tell fluent fabrications from the truth at all. Their ability came back once the interface showed how many claims had been verified Can readers tell truth from fabrication without evidence signals?. So the problem isn't really laziness. Without an outside signal, close reading of fluent text doesn't reveal much. One framework calls this 'scaled System 1': people take fast, intuitive fluency as a stand-in for careful reasoning, and that combines with confirmation bias to push trust steadily off course Why do people trust AI outputs they shouldn't?.

The corpus also points to a cause on the model side. Preference training (RLHF) rewards confident single answers over clarifying questions, which cuts the back-and-forth checks that would invite scrutiny to 77.5% below human levels Does preference optimization harm conversational understanding?. A different set of notes changes the question itself. If an explanation's effect depends on who presents it, how it's framed, and who receives it, then reading it as a competence signal is a social reaction, not a failed close reading What if XAI is fundamentally a communication problem?. In everyday human conversation, explanations work because they're built together, through questions and checks for understanding What makes explanations work in real conversation?.

The part you might not expect is that the fix isn't telling people to read more carefully. It's changing what the explanation does: show both sides, display verification signals, or build in back-and-forth. A model-training result points the same way. Models learn more deeply from critiquing flawed answers than from copying correct ones Does critiquing errors teach deeper understanding than imitating correct answers?, which suggests people engage more when there's something to push against than when they're handed a polished conclusion.