Doctors got a helpful AI prediction but still didn't make better calls — so what went wrong in how they used it?

This explores why handing experts an AI prediction doesn't automatically make them better, and what goes wrong when they fold that prediction into their own judgment. The clearest evidence comes from an experiment with professional radiologists. On average, AI predictions did not improve their diagnoses, even though the AI on its own carried useful information Why don't radiologists benefit from AI predictions?. The AI was useful. The losses came from how the radiologists used it, and there were two separate mistakes.

The first mistake is underweighting. Radiologists gave the AI's output less influence than it deserved, so they left accuracy unused. The second mistake is less obvious: they treated their own read of the image as independent of the AI's read. Both are looking at the same X-ray. When they agree, that isn't two independent votes, and when they disagree, the disagreement means something different than it would between two unrelated sources. Combining two correlated signals as if they were independent gives you the wrong confidence level in either direction. So the question isn't simply whether experts trust AI enough. It's whether they have a correct sense of how the AI's evidence relates to their own.

The same lopsided updating shows up on the machine side. LLM agents learning in context update their beliefs asymmetrically: they are optimistic about outcomes of actions they chose and pessimistic about the alternatives they passed over. The bias disappears when the task isn't framed around the model's own choices Do language models learn differently from good versus bad outcomes?. That pattern looks a lot like an expert favoring their own reading over an outside signal, and it suggests that weighting your own judgment above others' may be a general property of learners acting on their own choices, not a quirk of radiologists. The opposite failure exists too. Users can over-trust fluent AI output when confusing a model's output with reality, mistaking quick intuition for reasoning, and confirmation bias stack on top of each other Why do people trust AI outputs they shouldn't?. Put the two together and there's no single safe default. Experts can lose by trusting AI too little or too much, depending on the setting.

That points to a fix that's easy to miss: the AI has to give experts something they can weight correctly. A bare prediction doesn't do that. A confidence score is only useful if it's honest, and standard right-or-wrong reward training pushes models toward confident guessing that breaks this link. Adding a penalty for confident wrong answers restores it Does binary reward training hurt model calibration?. Another approach grounds a model's confidence in its own track record: how often it was right in past cases where it felt similarly sure Can past performance predict when a model will be right?. That is the kind of information a radiologist would need in order to stop treating the AI as either a rival or an oracle.

The takeaway is that better human-AI decisions depend less on making the AI more accurate and more on helping people combine signals correctly: knowing when the AI's view overlaps with their own and when it adds something new. The corpus documents the problem well in one medical domain. It has less direct evidence on interventions that teach experts to weight AI correctly, so that remains an open question rather than a solved one.