Why do rare cases in medicine and science require models that preserve tail distributions?
This explores why methods that sharpen or collapse a model's probability distribution—the very thing that boosts average accuracy—are exactly what break down on the rare diseases and novel results that live in the tails.
This explores why preserving the tail of a distribution matters for rare cases: the rare disease, the unusual clinical presentation, the experimental result nobody expected. These cases live in the low-probability part of a model's distribution, and the corpus shows that many of our standard training and inference recipes quietly flatten that tail in pursuit of better average performance.
The sharpest example is outcome-based reinforcement learning. When you reward only the final correct answer, the model concentrates probability mass on the trajectories that already work—and the corpus shows this diversity loss doesn't stay local. It transfers from solved problems to unsolved ones, globally narrowing the space of strategies the model will even consider Does outcome-based RL diversity loss spread across unsolved problems?. A rare case is precisely the kind of unsolved problem that needs an off-distribution strategy, so a model trained this way arrives already blind to it. The same flattening happens at inference time in a subtler way: setting temperature to zero feels like 'reliability,' but it just replays a single draw from the distribution over and over. Consistency is not coverage, and the tail you collapsed is the tail you can no longer sample from Does setting temperature to zero actually make LLM outputs reliable?.
What makes this dangerous in medicine and science specifically is that tail collapse pairs with overconfidence. Models trained on general text underperform badly on specialized clinical inference—yet report high confidence anyway, and prompting tricks that fix general accuracy don't dent that overconfidence Why do language models fail confidently in specialized domains?. A flattened distribution hides exactly the signal a clinician needs: 'this is rare, I'm uncertain, defer.' One promising fix is to make rarity itself a first-class trigger. Internal confidence and data-rarity signals catch different failures—confidence misses hallucinations about rare entities, while rarity catches the case the model has barely seen—so combining them outperforms either alone Should RAG systems use model confidence or data rarity to trigger retrieval?.
The constructive thread in the corpus is about keeping the distribution wide on purpose. Stochastic latent reasoning lets a model hold a distribution over solutions rather than committing to one, which is what you want when a problem is genuinely ambiguous or admits several valid answers Can stochastic latent reasoning help models explore multiple solutions?. Staying close to the base model—low KL drift—preserves the plasticity to keep learning new tasks instead of stalling when the domain shifts, a structural argument for not over-sharpening during adaptation Does staying close to the base model preserve learning ability?. And there's a real cost to ignoring this: training on near-impossible samples makes the model treat its rare accidental successes as high-value, reinforcing degenerate shortcuts that then contaminate skills it already had Do overly hard RLVR samples actually harm model capabilities?.
The twist worth carrying away is that the tail isn't only error—it's also discovery. The same pattern-integration tendency that we label 'hallucination' on a backward-looking retrieval task becomes genuine prediction on a forward-looking one: fine-tuned models out-predict human neuroscientists on which experimental results actually occurred Can LLMs predict novel scientific results better than experts?. A rare scientific result and a hallucination are both low-probability completions; a model that has flattened its tail can produce neither the dangerous error nor the useful novelty. Preserving the tail is what keeps both rare-case caution and rare-case insight on the table.
Sources 8 notes
RL that rewards only final answer correctness sharpens the policy globally, concentrating probability mass on correct trajectories for solved problems while simultaneously reducing diversity on unsolved ones. Historical exploration (training diversity via UCB-style bonuses) and batch exploration (test-time diversity via repetition penalties) require structurally different mechanisms.
Fixed seeds and zero temperature replicate the same output repeatedly, but that output remains one draw from the model's probability distribution. McDonald's omega testing across 100 repetitions reveals that consistency does not equal reliability.
LLMs trained on general text lack sufficient exposure to domain-specific examples, leading to low accuracy paired with high confidence in clinical NLI tasks. Prompting techniques that improved general performance fail to reduce overconfidence in specialized domains.
Model confidence and data-rarity signals catch orthogonal failure modes: confidence misses hallucinations about rare entities, while rarity misses uncertain reasoning about common knowledge. Hybrid triggers substantially outperform either signal alone.
GRAM replaces deterministic latent updates with stochastic sampling, enabling models to represent distributions over solutions rather than single predictions. This allows handling of ambiguous problems and multiple valid strategies that deterministic designs cannot represent.
FST-trained models stay up to 70% closer to their base distribution than parameter-only RL, and this reduced drift preserves the model's ability to learn subsequent tasks effectively. Parameter-only approaches stall when task domains change, while low KL drift enables sustained adaptation.
Training on nearly-impossible problems causes models to learn degenerate shortcuts rather than genuine reasoning, and these shortcuts contaminate pre-existing capabilities. Group-relative normalization treats rare accidental successes as high-advantage trajectories, reinforcing answer repetition and computation-skipping instead of sound reasoning patterns.
BrainBench benchmarks show fine-tuned LLMs outperform neuroscience experts at predicting which experimental results actually occurred. The same pattern-integration tendency that causes hallucination in retrieval tasks enables genuine prediction in forward-looking scenarios.