Medical AI models look better than general ones, but under fair testing they significantly beat them on under 10% of tasks.

This explores which medical tasks really improve when a general model gets extra pretraining on medical text. The answer starts by questioning the premise. When seven medical LLMs were compared fairly with their own base models, each model got its own tuned prompts and the results were checked for statistical noise. Under those conditions the medical versions were significantly better on only 9.4% of tasks. Without those controls they looked better on 70.5% (Does medical pretraining actually improve model performance?). Medical vision-language models fell from 62.5% to 6.3%. The corpus doesn't name a group of medical tasks that clearly wins. Most of the apparent benefit came from comparing a well-prompted specialist against a poorly prompted generalist.

That result fits another finding. Prompting can't give a model knowledge it never learned. It can only bring out knowledge already stored somewhere in the model (Can prompt optimization teach models knowledge they lack?). So if good prompting closes most of the gap, the general models already knew most of the medicine. Medical pretraining mostly made that knowledge easier to reach. It didn't add much that was new. The places to look for real gains are therefore tasks where general training data is thin. One example is clinical inference: deciding whether one clinical statement follows from another. There, general models are often wrong and highly confident at the same time, and clever prompting doesn't reduce that overconfidence (Why do language models fail confidently in specialized domains?).

A more useful way to frame the question is to ask what kind of change a medical task needs. One line of work suggests that factual knowledge lives mostly in a model's lower layers and reasoning in its higher layers. That would explain why training a model to reason better helps with math but can hurt medicine, where the bottleneck is remembering facts correctly (Why does reasoning training help math but hurt medical tasks?). Reinforcement learning on medical reasoning seems to help mainly by getting the model to stop using wrong facts in its reasoning, not by teaching it new ones (Does RL improve domain reasoning by adding knowledge or removing it?). Other approaches reward the model for giving a sound explanation as well as the right answer, and these seem to build knowledge in more firmly than standard fine-tuning (Can reinforcement learning embed domain knowledge more effectively than supervised fine-tuning?).

There's also a hidden cost to watch for. Training directly on domain data can damage the very knowledge stored in those lower layers. Methods that adjust the model's outputs while generating text, leaving its weights untouched, keep that knowledge intact better (Can decoding-time tuning preserve knowledge better than weight fine-tuning?). Benchmark scores can also mislead. Supervised fine-tuning can raise final-answer accuracy while the reasoning behind each answer gets worse, with the model making up a justification after reaching the answer (Does supervised fine-tuning improve reasoning or just answers?). Every adaptation method has a narrow set of conditions where it works best, and its benefits often come with degradation that is hard to see (How do domain training techniques actually reshape model behavior?).

The surprising lesson is that "which medical tasks benefit?" may be less important than "was the comparison fair?" Many claims for specialist medical models fade once the general model gets the same care with prompting. Where gains do hold up, they are most likely on rare, specialized material the general model never saw. They are least likely on standard medical-exam questions that are already well covered on the open web.