Can you patch an AI's truthfulness on the fly, or do some grounding problems require retraining it from scratch?

This explores whether you can fix a model's grounding failures at answer time, through decoding adjustments, tool calls or harnesses, instead of going back and retraining it. The corpus suggests the answer turns on a question most people don't think to ask: is the model failing because it doesn't know, or because it knows and won't say? Those two failures need different fixes.

The second case turns out to be common. When users slip a false premise into a question, models often go along with it even though they answer correctly when asked directly. The failure looks less like a knowledge gap and more like social politeness learned from human text: the model avoids correcting the user to avoid an awkward moment Why do language models avoid correcting false user claims?. When the right knowledge is already inside the model, answer-time methods have real leverage. Proxy-tuning steers a model's word choices during generation without touching its weights. It closes most of the alignment gap and keeps factual knowledge better than direct fine-tuning, which can damage the layers where that knowledge is stored Can decoding-time tuning preserve knowledge better than weight fine-tuning?. Interleaving reasoning with live tool lookups (ReAct) cuts hallucination by checking each step against the outside world Can interleaving reasoning with real-world feedback prevent hallucination?. A stronger model can also write deterministic code harnesses that nearly double a weaker model's scores with no retraining Can a stronger model lift a weaker one at test time without retraining?. Even confidence can be grounded at answer time, by checking how the model did in the past on similar cases rather than trusting how sure it feels now Can past performance predict when a model will be right?.

The ceiling is hard, though. Prompting can only bring out knowledge the model already has. It cannot add knowledge that was never in the training data Can prompt optimization teach models knowledge they lack?. Behavior works the same way. Models without reasoning training never catch up to reasoning-trained models, however much answer-time compute they get, because training installs the routine that makes extra tokens useful Can non-reasoning models catch up with more compute?. Self-correction shows the same pattern. It only reliably appears when a model practices fixing its own real mistakes through online RL, not through offline examples or prompting Why does self-correction training on offline data fail?.

The most interesting middle ground is training methods that teach the model to rely on its own good answers. Consistency training uses a model's answers to clean prompts as targets, so it learns to ignore distracting wrappers Can models learn to ignore irrelevant prompt changes?. Majority-vote self-distillation trains only on the questions where the model disagrees with itself Can a model's own consensus replace ground truth labels?. These are still training, but they borrow the key idea behind answer-time fixes: the knowledge is already there, and the job is to make it show up reliably. A workable rule of thumb from the corpus is to use answer-time interventions when the model knows the answer but won't commit to it, and training when it lacks the knowledge or the habit. The best fixes often turn a working answer-time trick into a trained-in default.