Do LLM explanations actually help people judge answer quality?

A think-aloud study (16 participants) and a preregistered, controlled experiment (N=308) examined which features of a large language model's response shape how much people rely on it when answering objective questions. The experiment varied three features in a 2×2×2 within-subjects design — accuracy of the LLM's answer, presence of an explanation, and presence of clickable sources — in responses from a hypothetical LLM called "Theta," built in advance from real ChatGPT and Perplexity AI outputs so the features could be fully controlled. The central finding: "the presence of explanations increases reliance on both correct and incorrect responses," while "we observe less reliance on incorrect responses when sources are provided or when explanations exhibit inconsistencies."

The authors attribute the explanation effect to superficial cues of quality rather than engagement with content: prior psychology work shows explanations "are often found compelling even when they contain little content... or content that experts judge irrelevant," especially "when time and prior knowledge are limited." In the absence of effort and expertise, people fall back on cues like fluency — "a characteristic that LLM explanations typically possess in spades." Sources work differently: the authors suggest they "encouraged participants to engage in slow and careful System 2 thinking, instead of quick and automatic System 1 thinking," evidenced by participants spending "significantly more time on task when provided with sources, especially when the LLM's answer was incorrect," and reporting that they switched to a different answer when sources conflicted with Theta's claim. Inconsistencies worked as a third, independent unreliability cue: in the think-aloud study, participants "interpret inconsistencies in explanations... as a cue of unreliability."

This gives explanation-driven overreliance a controlled, causal demonstration rather than a correlational one. It lines up with Does longer explanation text actually improve answer evaluation?, which found explanation length inflates confidence without improving accuracy discrimination in a quiz setting — together the two studies suggest explanations raise trust through sheer presence and length, independent of whether the content is right. It also extends Do users worldwide trust confident AI outputs even when wrong? beyond confidence language to a second class of surface feature — explanation presence — that drives reliance regardless of accuracy, while adding sources and inconsistency-detection as two of the few levers in this literature shown to push reliance in the appropriate direction rather than the overconfident one.

The authors are explicit that both studies were confined to objective question-answering and "may not generalize to other contexts of LLM use (e.g., writing, idea generation, and task automation)." The sources tested "were all real and tended to provide accurate and relevant information" — the paper notes this is "not always the case" for production systems, where sources can be fake or unsupported, in which case the protective effect observed here may not hold or could even reverse. Study 1's think-aloud format also may have changed behavior (63.8% source-clicking versus 25.8% in the online Study 2), so the natural-setting estimate of how much people seek out sources is likely higher than in ordinary unobserved use.