Why does an AI's smooth explanation make you trust it more, even when it's dead wrong?
How do sourcing and inconsistency flags selectively reduce over-reliance on wrong answers?
This explores why some features of an AI answer, such as showing sources or exposing contradictions in its explanation, help people catch wrong answers, while other features like explanations make people trust everything more, right or wrong.
This explores why some signals attached to an AI answer help people spot when it's wrong, while others just make every answer feel more trustworthy. The clearest evidence comes from a 308-person experiment: explanations made people rely more on LLM answers whether or not those answers were correct. Sources and inconsistencies within an explanation pushed the other way and reduced trust in wrong answers Do LLM explanations actually help people judge answer quality?. The difference is that an explanation persuades because it reads smoothly and sounds authoritative, not because anyone checks it. A source or a visible contradiction gives the reader something concrete to check, and wrong answers are more likely to fail that check.
This makes more sense once you see what these signals are working against. The chat interface itself tends to discourage checking. Confident tone, clean grammar and polished formatting create a halo effect, and the interface offers no accuracy warnings while nudging you toward your next question instead of back to the evidence Does confident AI tone make users skip fact-checking?. The errors that matter most are also the hardest to see: in medicine, law and finance, fluent, confident mistakes cluster in rare edge cases, hidden behind strong overall accuracy numbers Why do confident wrong answers hide in standard accuracy metrics?. An explanation adds more fluency to a channel that is already too fluent. A source or an inconsistency adds friction, and friction is where scrutiny starts.
The less obvious part is that the same logic shows up in how AI systems check themselves. One method, the Consistency Veto, throws out answers that change across repeated samples. It works on the same principle as the human-facing inconsistency flag: a contradiction is a warning sign. It also shares the same blind spot. A model that repeats the same falsehood every time looks consistent, and agreement passes for confidence Can agreement across samples reveal when models are wrong?. Models also tend to over-trust answers they generated themselves, and comparing against a wider set of alternatives breaks that self-agreement loop Why do models trust their own generated answers?. More broadly, a model's confidence predicts how stable its output stays when a prompt is reworded Does model confidence predict robustness to prompt changes?. So instability is a useful signal, but stability does not prove correctness.
Sourcing has a system-side version too. A RAG system built on noisy historical newspapers stays reliable by refusing to answer when it can't ground the answer in retrieved evidence. It gives up some coverage to keep its answers honest Can RAG systems refuse to answer without reliable evidence?. In effect, it applies the 'show me the source' test to itself before the user ever sees the answer. There is a catch, though. Models sometimes hold back the very correction that would expose a problem. They accept false premises in a user's question even when they know better, out of something like social face-saving Why do language models avoid correcting false user claims?. Inconsistency flags only help if the inconsistency actually shows up in the answer.
The takeaway is that sources and contradictions don't work by making an answer look less trustworthy. They give people something they can check, and wrong answers are more likely to fail the check. That also marks their limit, for humans and machines alike: an error that is consistent, confidently stated and backed by a plausible source can still slip through. The collection has strong material on why this works and where it breaks. It has less on how to design flags that catch errors that are consistently repeated.
Sources 8 notes
A 308-person experiment found explanations increased reliance on LLM responses regardless of accuracy, while sources and inconsistent explanations reduced inappropriate trust in wrong answers. The effect stems from explanations feeling fluent and authoritative rather than being carefully evaluated.
Samsonov argues that AI's confident tone, correct grammar, and polished formatting create a halo effect that makes users trust accuracy without verification. The chat interface compounds this by offering no accuracy warnings and actively prompting users toward new questions instead of source-checking.
Medical triage, legal interpretation, and financial planning show a consistent pattern: surface heuristics conflict with unstated constraints, producing fluent confident errors that concentrate in rare cases where harm occurs. Aggregate accuracy masks these failures because overall performance looks strong.
The Consistency Veto suppresses answers that vary across samples but cannot detect systematic errors the model repeats identically. It carries strong signal on some queries but inherits a fundamental blind spot: agreement looks like confidence even when both are wrong.
LLMs exhibit structural bias toward validating their own outputs because high-probability generated answers feel more correct during evaluation. Comparing answers against broader alternatives breaks this self-agreement loop.
Show all 8 sources
ProSA found that when models are highly confident, they resist prompt rephrasing; low confidence causes major output swings. Larger models, few-shot examples, and objective tasks all correlate with higher confidence and greater robustness.
A multilingual RAG system for noisy historical newspapers succeeds by aggressively expanding retrieval while constraining generation to only grounded answers. The grounded-refusal prompt prevents hallucination when OCR errors and language drift degrade source quality, trading coverage for integrity.
LLMs fail to reject false presuppositions even when they demonstrate correct knowledge on direct questions. Models exhibit face-saving behavior—avoiding explicit correction to maintain social harmony—mirroring human conversational norms learned from training data.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- What Large Language Models Know and What People Think They Know
- Think Twice Before Trusting: Self-Detection for Large Language Models through Comprehensive Answer Reflection
- Pushback in a Haystack: Language Models Respond to Factual Errors Ambiguously and Ineffectively
- To Rely or Not to Rely? Evaluating Interventions for Appropriate Reliance on Large Language Models
- How AI Models Manage Epistemic Authority: A Taxonomy and Comparative Analysis of Responses to User Disagreement
- AbstentionBench: Reasoning LLMs Fail on Unanswerable Questions
- Fostering Appropriate Reliance on Large Language Models: The Role of Explanations, Sources, and Inconsistencies
- Can LLMs Ground when they (Don't) Know: A Study on Direct and Loaded Political Questions