Letting an AI explain its reasoning sounds helpful — but does it actually lead to better decisions, or just more misplaced trust?
Can AI explainability help humans and machines reach better joint decisions?
This explores whether explaining how an AI reaches its outputs actually helps a human and an AI decide better together, or whether something other than explanation does the real work.
This explores whether explaining how an AI reaches its outputs helps a person and an AI make better decisions together, or whether something else does the real work. The corpus's answer is surprising: explanations on their own often make joint decisions worse. A review of 35 studies found that explainable AI tends to deepen automation bias rather than reduce it. People see a plausible rationale, trust the system more and check its work less. Low AI-literacy users are hit hardest Does explaining AI decisions actually reduce user overreliance on it?. What actually helped was active engagement: getting people to verify and push back, rather than just giving them more transparency.
One reason explanations mislead is that they may not reflect what the model actually did. Supervised fine-tuning can raise a model's accuracy while making its step-by-step reasoning worse. The model reaches correct answers and then builds a rationale after the fact Does supervised fine-tuning improve reasoning or just answers?. Going further, a model can pass every test while its internal structure is incoherent, so a clean-looking explanation can sit on top of reasoning nobody would endorse Can AI pass every test while understanding nothing?. When explanations are produced like this, they persuade people without informing them.
The more promising work changes what the AI hands over rather than how well it explains itself. In 'Learning to Guide', the machine doesn't make a decision and justify it. It points out which parts of the input deserve attention and leaves the call to the human. This removes the anchoring effect, where people latch onto the AI's verdict Can AI guidance reduce anchoring bias better than AI decisions?. A different approach skips explanation entirely. When humans and AI make independent predictions and the results are combined, the pair beats either one alone, as long as they make different kinds of mistakes Can humans and AI improve decisions by predicting together?. The surprising implication is that too much explanation could hurt here: if the human absorbs the AI's reasoning, their errors start to match the AI's, and the benefit of combining two different judgments disappears.
That moves the question from interface design to organization design. Puranam treats a human-AI pair as a small organization. The key choices are who handles which kind of task and how each side learns from feedback, and those choices shape outcomes more than how transparent any single output is How should humans and AI systems divide decision-making tasks?. Seen this way, an explanation is one channel among several, and it helps only when the structure around it asks the human to think rather than approve.
There's also a time limit on all this. Leike argues that today's alignment successes depend on models still being understandable to the people overseeing them. Once models act in ways people can't follow, that advantage is gone Can we solve AI alignment before models become uninterpretable?. So explainability matters most as a tool for oversight while it still works, and the corpus suggests that for everyday joint decisions, keeping human judgment independent beats making the AI more persuasive.
Sources 7 notes
A systematic review of 35 studies (2015–2025) found that XAI explanations frequently reinforce misplaced trust rather than curb over-reliance, especially among low-AI-literacy users. Active user engagement and critical verification, not explanation transparency alone, effectively counters automation bias.
Supervised fine-tuning improves final-answer accuracy on benchmarks but cuts Information Gain by 38.9 percent, meaning models generate correct answers through post-hoc rationalization rather than genuine inferential steps. Standard metrics miss this degradation because they only measure final correctness.
The Fractured Entangled Representation hypothesis shows that SGD-trained networks can produce identical outputs across all inputs while maintaining radically different internal representations. Standard benchmarks cannot detect this structural difference.
Learning to Guide eliminates anchoring bias and unassisted hard cases by having machines supply interpretive guidance rather than autonomous decisions, keeping responsibility with humans while improving their judgment through enhanced perception.
Choudhary et al. show that aggregating human and AI predictions on identical tasks outperforms either alone whenever their errors differ sufficiently, even if neither achieves high accuracy individually. This requires diversity in how each errs, not specialization.
Show all 7 sources
Puranam frames human-AI pairs as multi-agent organizations requiring design choices about task division and learning configuration. He distinguishes three task types and shows learning mechanisms produce different performance outcomes based on feedback structure and decision architecture.
Leike reports that simple interventions reduced agentic misalignment to near zero in recent models through automated auditing metrics, but this success depends on human interpretability; once models act in ways humans cannot understand, alignment becomes an unsolved hard problem.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- To Trust or to Think: Cognitive Forcing Functions Can Reduce Overreliance on AI in AI-assisted Decision-making
- Complementarity in Human-AI Collaboration: Concept, Sources, and Evidence
- Exploring automation bias in human-AI collaboration: a review and implications for explainable AI
- Human-AI Ensembles: When Can They Work?
- Can AI Explanations Make You Change Your Mind?
- Can AI Do Strategy?
- Exploring the Impact of Artificial Intelligence: Prediction versus Judgment
- Human-AI collaborative decision-making as an organization design problem