Why is editing specific facts so difficult in language models?
This reads the question as asking why you can't simply reach in and change one fact a model 'knows' — and the corpus answers obliquely, because facts in an LLM aren't stored in editable slots but baked into overlapping training associations.
This explores why editing a specific fact in a language model is so hard, and the collection's best answer is that there's no single place where a fact lives to be edited. What looks like a discrete fact is really a strong statistical association laid down during training, and those priors dominate. Research shows that even when you put corrected information directly in the prompt, the model often ignores it — parametric knowledge from training overrides the in-context correction, and only intervening in the model's internal representations reliably changes the output Why do language models ignore information in their context?. So the easiest 'edit' — just tell it the new fact — frequently fails.
The next instinct is to fix the fact by prompting cleverly, but the corpus draws a hard line here: prompting only reorganizes knowledge already in the training distribution; it cannot inject knowledge the model never learned Can prompt optimization teach models knowledge they lack?. That means surface-level interventions can surface or suppress what's there, but they don't write new facts in. Editing a fact is closer to retraining than to rewriting a database entry.
There's also a deeper reason the target is slippery: the model doesn't hold one committed version of things. Shanahan's 20-questions test shows an LLM maintains a superposition of consistent possibilities and samples one at generation time, so regenerating the same prompt yields different answers Do large language models actually commit to a single character?. If the model never commits to a single stored value, there isn't a fixed object to overwrite — you'd have to shift a whole distribution.
The corpus also hints at why some facts are harder to touch than others: representation strength tracks how often something appeared in training. Models reason worse about historical legal cases precisely because older precedent is under-represented and stored more shallowly Why do language models struggle with historical legal cases?. Strong, frequently-seen facts are entrenched and resist change; weak ones are diffuse and hard to locate. And whatever you change can ripple — work on RLVR shows learning concentrates in a small set of high-entropy 'forking' tokens, suggesting the levers that actually move behavior are few and load-bearing Do high-entropy tokens drive reasoning model improvements?.
The thing you didn't know you wanted to know: 'fact editing' fails not because the technique is immature but because the premise is wrong. There's no editable fact-cell — only entangled associations, sampled distributions, and depth that mirrors training frequency. Even a model's ability to fix itself is formally bounded without external grounding What stops large language models from improving themselves?, which is why durable correction tends to require representation-level surgery or retraining rather than a quick patch.
Sources 6 notes
Research demonstrates that LMs generate outputs inconsistent with their context because parametric knowledge from training dominates over in-context information. Textual prompting alone cannot override strong priors; causal intervention in representations is required.
Prompting works entirely within a model's pre-existing training distribution and cannot supply domain knowledge absent from training data. This creates a hard ceiling: no prompt strategy can compensate for missing foundational knowledge, only reorganize what already exists.
Shanahan's 20-questions test shows LLMs maintain a superposition of consistent objects or characters and sample from that distribution at generation time. Regenerating the same response yields different outputs, each consistent with prior context, proving no fixed commitment exists.
Supreme Court overruling benchmark (236 pairs) reveals era sensitivity: models perform worse on historical cases than modern ones. Root cause is training corpus over-representation of recent cases, creating shallower representations of older precedent.
Only ~20% of tokens exhibit high entropy as pivotal reasoning decision points; RLVR primarily adjusts these forking tokens. Training exclusively on them matches or exceeds full-gradient performance, revealing that the minority carries the learning signal.
Self-improvement in LLMs is formally bounded by the generation-verification gap, meaning every reliable fix requires something external to validate and enforce it. Models cannot escape this constraint through metacognition alone.