Should we judge an AI's explanation by how well it helps you catch its mistakes, not by how accurately it describes itself?
How should AI explanations be evaluated as human interfaces rather than model properties?
This explores what it means to judge an AI explanation by what it does to the person receiving it (whether it helps them decide, catch errors, or contest a result) instead of by how faithfully it describes the model's internals.
This explores what it means to judge an AI explanation by what it does to the person receiving it, whether it helps them decide, catch errors, or contest a result, instead of by how faithfully it describes the model's internals. The corpus suggests that a good explanation isn't a property the model has. It's an outcome that shows up, or doesn't, in a particular human situation.
Start with the most direct evidence. Reasoning traces and post-hoc explanations make people *more* likely to accept an AI's answer whether or not it's right, which is false trust rather than understanding. In one study, only contrastive dual explanations, which lay out arguments for and against an answer, actually helped users tell correct outputs from incorrect ones (Do explanations actually help users spot AI mistakes?). A model-side score such as "is this explanation faithful or fluent?" would have rated all of these explanations well. The human-side test, whether people catch more mistakes, separates them cleanly.
The corpus gives two reasons a model-property view falls short. One is that explanation quality depends on a triad of who presents it, how it's framed, and who receives it. Evaluations that ignore this triad measure only a narrow slice of real-world effectiveness (What if XAI is fundamentally a communication problem?). The other is more provocative: explanations often work as adoption arguments dressed up as technical descriptions. They describe how the system works and also persuade you it deserves use, and the persuasion borrows credibility from the description (Are AI explanations really descriptions or adoption arguments?). If so, an interface-level evaluation has to ask what the explanation is *doing* to the reader, not only whether it's accurate. It goes further still: meaning may be settled at the level of social groups, through layered interpretations of interpretations, rather than inside a single user-AI exchange. Lab-tested explanations stripped of that social context may not predict real-world effectiveness (Where does the meaning of an AI explanation actually come from?).
Other notes suggest what a human-centered design might look like. Formal argumentation frameworks turn an output into a graph of attacks and defenses, so users can pinpoint and contest the specific premise they reject, which unstructured text doesn't allow (Can formal argumentation make AI decisions truly contestable?). Conversation-analysis ideas point the same way: insert-expansions, the clarifying questions people ask mid-conversation, offer a model of when an agent should check in with the user, which prevents misunderstanding rather than repairing it afterward (When should AI agents ask users instead of just searching?). And because AI context is mutable and ephemeral, unlike the fixed layout of a conventional interface, users can't build stable habits around it. That makes explanation design closer to context engineering than to screen design (How does AI context differ from conventional software context?).
The collection has a gap. The measurement-side notes, such as reasoning fidelity through traceability and counterfactual adaptability (Can we measure reasoning quality beyond output plausibility?) and agent-based judging (Can agents evaluate AI outputs more reliably than language models?), tell us how to evaluate the model's reasoning or its outputs, not how a person experiences an explanation. Those are the model-property tools. The corpus points to what the human-interface tests should measure (error detection, calibrated trust, ability to contest) but doesn't yet offer a standard method for measuring them in realistic social settings.
Sources 9 notes
Reasoning traces and post-hoc explanations increase user acceptance of AI answers regardless of correctness, engendering false trust. Only dual explanations presenting arguments for and against the answer genuinely help users distinguish correct from incorrect outputs.
Explanation quality is not intrinsic to the explanation itself but depends on the rhetorical situation: who presents it, how it is framed, and what role the recipient plays. Evaluations that ignore this triad measure only a narrow slice of real-world effectiveness.
The Rhetorical XAI paper shows that explanations serve dual purposes: describing how AI works and justifying why it should be used. This rhetorical work has been hidden under transparency language, allowing adoption arguments to inherit credibility from behavioral descriptions.
Drawing on Luhmann's multi-layer cybernetics, AI explanation meaning is constituted at the social-group level through layered observations of observations, not produced inside dyadic human-AI dialogue. Lab-tested explanations stripped of social context will not predict real-world effectiveness.
Dung-style argumentation structures AI outputs as traversable attack/defense graphs, allowing users to identify and contest specific premises. Standard LLM outputs lack this structure, making it impossible to pinpoint which claims users actually reject.
Show all 9 sources
Tool-enabled LLMs drift from user intent through silent tool chaining. Conversation analysis reveals insert-expansions—clarifying intent, scoping responses, enhancing appeal—as a formal framework for proactive user consultation that prevents misunderstanding instead of recovering from it.
AI interactions operate on a substrate of constantly shifting context—prompt, history, retrieved data, hidden state—that users cannot internalize like traditional UIs. This structural mutability demands a new design discipline centered on context engineering rather than interface design.
Research identifies traceability, counterfactual adaptability, and motif compositionality as testable measures of human-like reasoning. These structural properties reveal whether an agent genuinely reasons causally or merely mimics coherent speech.
Eight-module agentic evaluation achieved 0.27% judge shift versus 31% for LLM-as-a-Judge on complex tasks. However, the memory module cascaded errors, revealing that agentic systems need error isolation mechanisms to maintain gains.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Expanding Explainability: Towards Social Transparency in AI systems
- Rhetorical XAI: Explaining AI’s Benefits as well as its Use via Rhetorical Design
- Evaluating the False Trust Engendered by LLM Explanations
- An Enigma of Artificial Reason: Investigating the Production-Evaluation Gap in Large Reasoning Models
- Can AI Explanations Make You Change Your Mind?
- Can We Trust AI Explanations? Evidence of Systematic Underreporting in Chain-of-Thought Reasoning
- “Understanding AI”: Semantic Grounding in Large Language Models
- Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate