LLMs can be badly wrong and still totally confident in specialized fields like medicine — and better prompting doesn't fix that.
How persistent is LLM overconfidence in specialized knowledge domains?
This explores whether LLMs stay overconfident when they work in specialized fields like medicine, and whether the usual fixes, such as better prompting, actually make that overconfidence go away.
This explores whether LLMs stay overconfident in specialized fields like medicine, and whether the usual fixes make that go away. The short answer from the corpus is that it's quite persistent. The most direct evidence comes from clinical inference tasks. Models trained mostly on general text have seen too few domain examples, so they end up with low accuracy and high confidence at the same time. Prompting techniques that improve general performance don't reduce that overconfidence in the specialized setting Why do language models fail confidently in specialized domains?. Better accuracy and better calibration turn out to be separate problems, and prompting only helps with the first. The corpus has one study aimed squarely at this question, so treat it as a strong signal rather than a settled map of every domain.
Why doesn't the model simply notice it's out of its depth? Because its sense of what it knows is shallow. Models can describe some of their own behaviors, but their self-reports are unstable, and they shift their beliefs under conversational pressure How well do language models understand their own knowledge?. A stranger version of the same gap is 'Potemkin understanding.' A model can correctly explain a concept, fail to apply it, and then recognize that it failed. That suggests explaining and doing run on largely disconnected tracks Can LLMs understand concepts they cannot apply?. In a specialized domain, a fluent explanation can therefore cover for a broken application. Turning the temperature down to zero won't expose this either. It just gives you the same possibly wrong answer every time, which is consistency and not reliability Does setting temperature to zero actually make LLM outputs reliable?.
The overconfidence also passes to the people reading the output. In experiments, readers' confidence followed the tone of the model's explanation, not the model's own internal probability of being right. Default explanations left people consistently overconfident. The gap narrowed only when explanations were deliberately written to reflect the model's real uncertainty Why do readers trust LLM explanations more than they should?. A related problem is that models can't tell a real expert's argument from a common assumption. They only see text, not the reputation and track record that give expertise its weight Can language models distinguish expert arguments from common assumptions?. Specialized fields depend heavily on knowing whose claims to trust, so this blind spot matters most there.
The most promising fix in the corpus comes from outside the model. Instead of asking the model how sure it feels, XConf looks up the model's past attempts at similar confidence levels and checks how often they were actually right. It matches the reliability of sampling ten answers at a tenth of the cost, and the improvement comes entirely from the stored track record Can past performance predict when a model will be right?. This fits a broader pattern: reliable agents move burdens like memory and verification into the system around the model rather than trusting the model to handle them Where does agent reliability actually come from?.
There's a tension worth knowing about. Some training methods now use the model's own confidence as the reward signal, which lets reinforcement learning reach domains that don't have answer checkers Can model confidence alone replace external answer verification?. If that confidence is least trustworthy in exactly the specialized domains these methods hope to reach, then rewarding it could reinforce the overconfidence instead of fixing it. The corpus doesn't test that directly, but it's the question these notes together point toward.
Sources 9 notes
LLMs trained on general text lack sufficient exposure to domain-specific examples, leading to low accuracy paired with high confidence in clinical NLI tasks. Prompting techniques that improved general performance fail to reduce overconfidence in specialized domains.
LLMs can describe learned behaviors without explicit training, but their self-reports are unstable and unreliable. Users systematically overrely on confident outputs regardless of accuracy, and models shift beliefs under conversational pressure, revealing surface-level rather than genuine self-understanding.
Models can explain concepts accurately, fail to apply them, and recognize the failure—a triple pattern incompatible with human cognition. This indicates functionally disconnected explanation and execution pathways rather than simple knowledge gaps.
Fixed seeds and zero temperature replicate the same output repeatedly, but that output remains one draw from the model's probability distribution. McDonald's omega testing across 100 repetitions reveals that consistency does not equal reliability.
Lab experiments on MMLU and TriviaQA show readers' confidence judgments track explanation language poorly compared to the model's internal probability estimates. The gap narrows when explanations are deliberately engineered to reflect internal confidence, but natural language explanations leave readers systematically overconfident.
Show all 9 sources
LLMs lose the social context that gives expert claims their force—reputation, track record, and standing—because they process only text, not the social world where expertise is built and evaluated.
XConf matches ten-sample self-consistency at a tenth of the cost by retrieving the model's past episodes with similar confidence levels and reading their historical success rates. Ablations show the signal depends entirely on stored outcomes, not on the retrieval prompt itself.
Research shows reliable LLM agents externalize three cognitive burdens—memory (state persistence), skills (procedural components), and protocols (structured interaction)—into a harness layer rather than relying on model scale alone. The harness unifies these externalities and eliminates the need for the model to solve the same problems repeatedly.
RLPR and INTUITOR successfully extend reinforcement learning for reasoning to general domains by using the model's own token probabilities and confidence levels as reward signals, eliminating the need for external verifiers or reference answers.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Post-Training Large Language Models via Reinforcement Learning from Self-Feedback
- What Large Language Models Know and What People Think They Know
- Beyond Accuracy: Evaluating the Reasoning Behavior of Large Language Models -- A Survey
- Fostering Appropriate Reliance on Large Language Models: The Role of Explanations, Sources, and Inconsistencies
- Word Meanings in Transformer Language Models
- Explain-Query-Test: Self-Evaluating LLMs Via Explanation and Comprehension Discrepancy
- Six misconceptions about large language models: A minimal model and diagnostic taxonomy
- Towards a Science of AI Agent Reliability