Can showing sources, or pointing out when an AI contradicts itself, stop people from trusting its wrong answers?
Can source citations or explicit inconsistencies reduce user overreliance on wrong answers?
This explores whether showing sources, or letting users see that an AI's explanation contradicts itself, helps people notice when an AI answer is wrong rather than trusting it anyway.
This explores whether design choices like attaching sources or exposing inconsistencies can stop people from leaning on AI answers that happen to be wrong. The most direct evidence in the corpus says yes, with a catch. In a 308-person experiment, sources and explanations that contradicted themselves both reduced people's trust in incorrect answers. Ordinary explanations backfired: they raised reliance on right and wrong answers alike, because a fluent explanation feels authoritative even when nobody checks it Do LLM explanations actually help people judge answer quality?. So the useful interventions are the ones that give the reader a reason to pause. A smooth justification does the opposite.
The catch is that sources only help when people actually read them. An analysis of 24,000 real search-chatbot comparisons found that irrelevant citations raised user preference almost as much as relevant ones Do users trust citations more when there are simply more of them?. Seen at a glance, a citation works as a trust badge, and more badges look more trustworthy. Put next to the experiment above, this suggests sources reduce overreliance only when they invite checking. A row of footnotes that nobody opens can make the problem worse.
This matters because the default chat experience pushes against checking. Confident tone, clean grammar and polished formatting create a halo effect. The interface also nudges users toward their next question instead of back to the evidence Does confident AI tone make users skip fact-checking?. Confident wrong answers are also hard to spot from the system side. They tend to cluster in rare, high-stakes cases while overall accuracy looks fine Why do confident wrong answers hide in standard accuracy metrics?. That puts more of the burden on the user at the moment they read the answer.
Inconsistency may help partly because models hide it. LLMs often go along with a user's false premise even when they know the correct fact, a face-saving habit picked up from human conversation Why do language models avoid correcting false user claims?. Models are also biased toward trusting their own answers, and comparing an answer against alternatives breaks that loop Why do models trust their own generated answers?. The same idea seems to work on people: showing competing or conflicting outputs breaks the illusion of one confident voice. On the system side, a related tactic is to have the system refuse when evidence is weak, instead of asking users to catch the errors Can RAG systems refuse to answer without reliable evidence?.
The corpus has only one direct user study on this exact question, so treat the conclusion as promising rather than settled. The takeaway you might not expect: a citation is not automatically a safeguard. It protects users only when it is relevant and they look at it, and a visible contradiction may protect them better than a polished explanation.
Sources 7 notes
A 308-person experiment found explanations increased reliance on LLM responses regardless of accuracy, while sources and inconsistent explanations reduced inappropriate trust in wrong answers. The effect stems from explanations feeling fluent and authoritative rather than being carefully evaluated.
Analysis of 24,000 Search Arena interactions shows irrelevant citations boost user preference (β=0.273) nearly as much as relevant citations (β=0.285), indicating citation count functions as a decoupled trust heuristic.
Samsonov argues that AI's confident tone, correct grammar, and polished formatting create a halo effect that makes users trust accuracy without verification. The chat interface compounds this by offering no accuracy warnings and actively prompting users toward new questions instead of source-checking.
Medical triage, legal interpretation, and financial planning show a consistent pattern: surface heuristics conflict with unstated constraints, producing fluent confident errors that concentrate in rare cases where harm occurs. Aggregate accuracy masks these failures because overall performance looks strong.
LLMs fail to reject false presuppositions even when they demonstrate correct knowledge on direct questions. Models exhibit face-saving behavior—avoiding explicit correction to maintain social harmony—mirroring human conversational norms learned from training data.
Show all 7 sources
LLMs exhibit structural bias toward validating their own outputs because high-probability generated answers feel more correct during evaluation. Comparing answers against broader alternatives breaks this self-agreement loop.
A multilingual RAG system for noisy historical newspapers succeeds by aggressively expanding retrieval while constraining generation to only grounded answers. The grounded-refusal prompt prevents hallucination when OCR errors and language drift degrade source quality, trading coverage for integrity.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Fostering Appropriate Reliance on Large Language Models: The Role of Explanations, Sources, and Inconsistencies
- Seeing to Think? How Source Transparency Design Shapes Interactive Information Seeking and Evaluation in Conversational AI
- Think Twice Before Trusting: Self-Detection for Large Language Models through Comprehensive Answer Reflection
- What Large Language Models Know and What People Think They Know
- LLM or Human? Perceptions of Trust and Information Quality in Research Summaries
- To Rely or Not to Rely? Evaluating Interventions for Appropriate Reliance on Large Language Models
- Conversational presentation mode increases credibility judgements during information search with ChatGPT
- Search Arena: Analyzing Search-Augmented LLMs