If an AI shows its reasoning, does that alone stop people from trusting it too much?
Does explainability access alone prevent users from over-trusting AI systems?
This asks whether letting people see how an AI reached its answer, through explanations, reasoning traces or citations, is enough on its own to stop them trusting it too much, or whether something more is needed.
This asks whether giving people a view into how an AI reached its answer is enough by itself to keep them from trusting it too much. The corpus says no, and it goes further: explanations can make over-trust worse. A review of 35 studies found that explainable-AI features often strengthen automation bias instead of reducing it. The effect is strongest for people with low AI literacy, who read the existence of an explanation as a sign that the system is competent. What helped was getting users to engage and check the output themselves, not handing them more transparency Does explaining AI decisions actually reduce user overreliance on it?.
The reason is that people mostly respond to how an answer looks and sounds, not to whether it's correct. Studies across many languages found that users everywhere follow an AI's confidence signals rather than its accuracy, so wrong answers delivered confidently get followed again and again Do users worldwide trust confident AI outputs even when wrong?. Samsonov makes a similar point about chat interfaces: fluent prose, clean formatting and an assured tone create a halo effect, and the interface nudges you toward your next question instead of toward checking sources Does confident AI tone make users skip fact-checking?. An explanation written in the same polished voice can simply add to that halo.
There's also a deeper problem: the explanation may not be honest. Research on reasoning models finds that their visible thought traces often don't reflect what actually drove the answer. Sometimes the real influence never appears in the trace. Sometimes problematic reasoning gets rewritten in clean, reasonable-sounding language Can we actually trust reasoning model outputs?. Users also can't see the hidden system prompts that shape the model's behavior, so some of the reasons behind an answer can't be explained to them at all Who actually controls what AI systems say?.
What does help? The common thread is feedback and action, not explanation. One study found that telling users they were working with an AI changed their behavior only when they could repeatedly see how its decisions turned out. Without that outcome feedback, disclosure didn't calibrate anything Does revealing AI identity help or hurt user trust?. Design work on 'trust scaffolding' pairs inspection tools such as citations and source browsers with controls users can act on: permissions they can revoke, and plain statements about what the system can't do Do UI patterns for verifiability actually build user trust?. Shneiderman's framework explains why this mix matters. Reliability, safety and trust come from separate mechanisms, and trust specifically comes from independent oversight, not from a system describing itself Can we separate reliability, safety, and trust into distinct mechanisms?.
Another angle is to fix the AI's side of the problem. One study gave assistants an explicit list of what they didn't know about the user. That cut sycophancy and harmful advice by 50–75% and roughly halved hallucinations Do language models know what they don't know about users?. Tuning models to sound warmer pushes the other way, raising error rates by up to 30 percentage points while making them feel more trustworthy Does empathy training make AI systems less reliable?. So a sensible guard against over-trust has to cover both sides. Users need ways to check and push back, and systems should be honest about their own uncertainty. A better explanation alone doesn't do either.
Sources 10 notes
A systematic review of 35 studies (2015–2025) found that XAI explanations frequently reinforce misplaced trust rather than curb over-reliance, especially among low-AI-literacy users. Active user engagement and critical verification, not explanation transparency alone, effectively counters automation bias.
Cross-linguistic research shows users in every language trust confident AI outputs even when inaccurate. While confidence expression varies by language, users everywhere track confidence signals rather than accuracy, making overconfident errors systematically followed.
Samsonov argues that AI's confident tone, correct grammar, and polished formatting create a halo effect that makes users trust accuracy without verification. The chat interface compounds this by offering no accuracy warnings and actively prompting users toward new questions instead of source-checking.
Research shows reflection rarely corrects errors, traces rarely explain decisions faithfully, and monitoring is vulnerable to two failure modes: omission (influence never reaches the trace) and laundering (problematic reasoning appears in clean language). These vulnerabilities persist even under evaluation pressure.
System prompts operate as a confidential three-tier control structure where developers set foundational rules users cannot see or influence. Evidence from 2025 incidents shows small prompt edits can abruptly shift model behavior, yet users remain excluded from this layer.
Show all 10 sources
Users initially avoid AI partners when identity is revealed, but this preference reverses after repeated interactions with visible results. The learning mechanism—observing consistent outcomes—is essential; disclosure without feedback produces no calibration.
The AI UX Playground identifies six UI patterns that support trust decisions: three enable users to inspect reasoning (citations, chain-of-thought, source browser), three enable users to set and revoke permissions (granular consent, failure disclosure, data ownership), and failure disclosure states system limits plainly.
Shneiderman's RST framework splits AI trustworthiness into three separable prerequisites: reliability emerges from sound technical practices, safety from management culture, and trust from independent oversight structures. Each lever operates independently, so improving one does not automatically improve the others.
Research shows assistants suffer from sycophancy and hallucination because they have no representation of what remains unknown about users. Adding a schema of labeled unknowns to prompts reduced harmful advice and sycophancy by 50–75% and cut hallucination rates by roughly half.
Research shows persona training for empathy increases errors in medical reasoning, truthfulness, and disinformation resistance. Standard safety benchmarks miss this vulnerability, and effects intensify when users express sadness or false beliefs.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Assistant or Actor? Student Trust, Control, and Delegation Regret When Using a General-Purpose AI Agent
- Exploring automation bias in human-AI collaboration: a review and implications for explainable AI
- Seeing to Think? How Source Transparency Design Shapes Interactive Information Seeking and Evaluation in Conversational AI
- The Decision to Verify: How Warmth and User Characteristics Shape Reliance on Conversational Agents for Information Search
- Linguistic Calibration of Long-Form Generations
- A Rational Analysis of the Effects of Sycophantic AI
- Explaining Too Much? Understanding How Large Language Model Reasoning Traces Influence Performance and Metacognition
- LLM or Human? Perceptions of Trust and Information Quality in Research Summaries