When AI medical advice is wrong, how much does it actually sway what patients and doctors decide to do?

This explores how wrong AI advice changes medical decisions for laypeople and for trained clinicians. The short answer: it changes them a lot, more than correct advice helps, and being suspicious of AI doesn't protect people as much as you'd expect. One caveat first. Most of these studies measure diagnostic accuracy or stated willingness to act. They don't track what treatment people actually chose afterward.

Start with patients. In a 300-person study, people picked out AI-written medical answers at chance level. They rated low-accuracy AI answers as valid and trustworthy enough to act on, at levels similar to or above real doctors' advice Can people tell AI medical advice from doctors' responses?. Clinicians did no better: they identified GPT-4 advice at chance and rated it as equally sound and more emotionally empathetic Can clinicians tell GPT-4 advice apart from expert advice?. So the first barrier is that people can't tell AI advice apart from a doctor's, and that includes the wrong advice.

What hits hardest is the asymmetry. In an ICU simulation, misleading AI predictions degraded nurse performance by 96–120%, while correct ones improved it by only 53–67%. Standard 'average accuracy' metrics hide this because they blend the gains and losses together Do wrong AI predictions hurt more than right ones help?. In mammography, wrong AI category suggestions dropped experienced radiologists from 82% to 45.5% accuracy. Inexperienced readers fell from nearly 80% to below 20% How much does wrong AI advice harm radiologist accuracy?. Experience softens the fall but doesn't prevent it. Among pathology experts, AI-induced errors happened in about 7% of assessments whether or not they were rushed. Time pressure didn't make these mistakes more frequent, but it made them worse: under the clock, experts leaned harder on the bad advice Does time pressure make AI advice more persuasive to experts?.

The counterintuitive finding is that distrust doesn't protect you. Radiologists rated advice lower when it was labeled as AI, yet their accuracy still tracked whether the advice was right or wrong, not where it came from Does labeling advice as AI change how clinicians use it?. In other words, people say they're skeptical and get pulled along anyway. A separate radiology experiment found the mirror image: on average, clinicians underweight AI predictions. They also wrongly treat their own read and the AI's as independent evidence, so they miss the gains from good predictions Why don't radiologists benefit from AI predictions?. Taken together, people get the worst of both: they don't benefit enough when the AI is right and still get dragged when it's wrong. The same pattern shows up outside medicine. When AI fact-checkers mislabel true headlines, people believe true news less, and that harm isn't offset elsewhere Does AI fact-checking actually help people spot misinformation?. Pushing back on the model doesn't reliably help either. When consultants challenged GPT-4, it doubled down on persuasion instead of admitting its limits Does validating AI output make models more defensive?.

What might help? One line of work suggests the trouble starts when AI hands over an answer to accept or reject, because the answer becomes an anchor. 'Learning to Guide' has the AI point out which parts of a case deserve attention instead of giving a verdict. That keeps the judgment with the human and removes the anchor that wrong answers exploit Can AI guidance reduce anchoring bias better than AI decisions?. So the design question may be less about making AI right more often and more about not giving people a ready-made conclusion to latch onto. The collection doesn't yet have studies following patients from bad AI advice to real-world treatment choices, and that gap is worth knowing about.