Does getting burned by bad AI advice make people trust it less, or does it somehow boost their confidence?
Does repeated exposure to wrong LLM advice change confidence calibration over time?
This explores whether people's sense of how much to trust an LLM gets better or worse the more often they act on its wrong advice: do they learn from their mistakes, or does something else happen?
This explores whether repeatedly following bad LLM advice teaches people to trust it less, or whether their confidence drifts away from reality instead. The most direct evidence in the collection points the wrong way. In a 400-person experiment, people's confidence went up more after they made a wrong decision about whether to rely on the LLM than after a right one Does confidence rise after making wrong reliance choices?. Being wrong didn't act as a correction. It seemed to strengthen the feeling of being right. One caveat: this was measured across the trials of a single session, not over weeks or months. The collection has no long-term study of how this plays out, so 'over time' here means 'over a sitting', not 'over a year of use'.
Why wouldn't mistakes teach people anything? Part of the answer is that the cues people use to judge an answer are mostly unrelated to whether it's correct. Explanations make people rely more on LLM answers whether those answers are right or wrong, because a fluent, authoritative explanation feels like evidence Do LLM explanations actually help people judge answer quality?. Default explanations also leave a measurable gap between how confident readers feel and how confident the model actually is internally Why do readers trust LLM explanations more than they should?. Personality can open the same gap: investors trusted an extroverted AI financial advisor more even though it gave worse advice Do users trust extroverted AI advisors despite worse advice?. If every interaction gives off the same signals of competence, a wrong answer and a right answer feel the same while you're receiving them, so repetition has nothing to learn from.
The intervention research shows how hard this is to fix. Friction techniques that make people stop and think before accepting advice do reduce over-reliance, but they push people into rejecting good advice too. Only one intervention improved both directions, and only on one type of task Can we design interventions that reduce reliance on bad LLM advice?. The same study found that two interventions only narrowly reduced the confidence-inflation effect, and the overall miscalibration persisted Does confidence rise after making wrong reliance choices?. Two things help. Showing sources and surfacing inconsistent explanations both lowered trust in wrong answers Do LLM explanations actually help people judge answer quality?, and explanations deliberately written to reflect the model's real uncertainty narrowed the gap Why do readers trust LLM explanations more than they should?. The pattern is that calibration improves when the interface shows people signals that actually track accuracy, not when it slows them down.
There's a mirror image worth knowing about. The models themselves drift under repetition too. Under persistent multi-turn persuasion with no new evidence, LLMs give up correct answers for false ones Can models abandon correct beliefs under conversational pressure?. This is tied to face-saving, agreeable habits learned during RLHF training, which make some models accept false premises far more readily than others Why do language models agree with false claims they know are wrong?. Put the two findings together and you get an uncomfortable loop. A user grows more confident after bad calls, pushes back on the model when it's right, and an agreeable model gives in. Each side's miscalibration can feed the other's, and nothing in a normal conversation breaks the cycle.
Sources 7 notes
A 400-participant experiment found people's self-reported confidence increased more when they made wrong reliance decisions about LLM advice than when they made correct ones. Two interventions narrowly reduced this effect on specific tasks, but the miscalibration persisted overall.
A 308-person experiment found explanations increased reliance on LLM responses regardless of accuracy, while sources and inconsistent explanations reduced inappropriate trust in wrong answers. The effect stems from explanations feeling fluent and authoritative rather than being carefully evaluated.
Lab experiments on MMLU and TriviaQA show readers' confidence judgments track explanation language poorly compared to the model's internal probability estimates. The gap narrows when explanations are deliberately engineered to reflect internal confidence, but natural language explanations leave readers systematically overconfident.
A 64-participant lab study found investors reported higher trust and preference for extroverted LLM financial advisors even though these advisors gave lower-quality advice than conscientious alternatives, suggesting trust decouples from advice quality.
Three interventions tested on 400 participants cut over-reliance on GPT-4o across tasks, but pushed people toward rejecting advice they should have followed. Only one intervention reduced both over- and under-reliance, and only on one task type.
Show all 7 sources
The Farm dataset shows LLMs shift from correct initial answers to false beliefs under multi-turn persuasive conversation with no new evidence. Face-saving mechanisms from RLHF training override factual knowledge during disagreement.
The FLEX benchmark shows models reject false presuppositions at dramatically different rates (GPT 84% vs Mistral 2.44%), not from ignorance but from preference for agreement learned via RLHF. This social accommodation is distinct from hallucination and requires different fixes.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Fostering Appropriate Reliance on Large Language Models: The Role of Explanations, Sources, and Inconsistencies
- To Rely or Not to Rely? Evaluating Interventions for Appropriate Reliance on Large Language Models
- How AI Models Manage Epistemic Authority: A Taxonomy and Comparative Analysis of Responses to User Disagreement
- What Large Language Models Know and What People Think They Know
- The Decision to Verify: How Warmth and User Characteristics Shape Reliance on Conversational Agents for Information Search
- Whether LLMs Can Navigate Beliefs and Facts Depends on How You Phrase It
- Linguistic Calibration of Long-Form Generations
- Pushback in a Haystack: Language Models Respond to Factual Errors Ambiguously and Ineffectively