What makes knowledge editing different from simply finding where facts are stored?
This explores the gap between *localizing* a fact inside a model (finding the weights or representations that hold it) and actually *changing what the model says* — and why those two things come apart.
This explores why knowledge editing is harder than the locate-then-overwrite picture suggests: pinning down where a fact lives in a model's representations is not the same as controlling whether that fact shapes what the model actually produces. The corpus's sharpest point here is that encoding and usage are separate processes. Language models routinely store facts in their internal representations while those same facts fail to causally influence generation Do language models actually use their encoded knowledge?. So even if you precisely identify the location of a fact, editing it there may move nothing downstream — you've found the storage without finding the lever.
Part of why facts are slippery to localize at all is *how* they got into the model. Factual recall depends on narrow, document-specific memorization — the model essentially leaned on particular source documents — whereas the reasoning that *uses* facts draws on broad, transferable procedural knowledge spread across many documents Does procedural knowledge drive reasoning more than factual retrieval?. That split matters for editing: a fact isn't a tidy entry in a lookup table, it's entangled with the procedures that retrieve and deploy it. Change the stored value and you may leave the retrieval habits untouched.
The corpus also hints that knowledge in these models is positional and structural, not atomic. StructTuning shows models learn *where* a piece of knowledge sits within a conceptual taxonomy — its relationship to neighboring concepts — rather than memorizing isolated text Can organizing knowledge structures beat raw training data volume?. If knowledge is held as position-within-a-structure, then editing one fact means perturbing a web of relationships, which is exactly why naive overwrites produce inconsistent or contradictory behavior.
There's a deeper framing worth pulling in: models that learn purely from data build representations nobody can cleanly read or surgically correct — the cost of tacit, data-only learning is uninterpretable internals where explicit fixes don't take Does refusing explicit knowledge harm AI system performance?. This is the flip side of the editing problem. The reason you can't just find-and-replace is the same reason the knowledge is powerful but opaque: it was never stored as discrete, addressable facts in the first place. Approaches that externalize knowledge into explicit, inspectable structures — like reasoning held in knowledge-graph triples — are partly an answer to this, trading some of the model's tacit fluency for the ability to actually see and revise what it 'knows' Can structuring reasoning as knowledge graphs help smaller models solve complex tasks?.
The takeaway you didn't know you wanted: 'finding where a fact is stored' assumes facts are stored *as facts*. The corpus suggests they're stored as causally-inert traces, document-specific memories, and positions in a conceptual web all at once — so editing is less like correcting a database row and more like nudging a system whose storage and behavior were never the same thing.
Sources 5 notes
Multiple studies confirm that language models can encode facts in their representations while those facts fail to causally affect downstream outputs. Encoding and usage are distinct processes.
Analysis of 5 million pretraining documents shows reasoning relies on broad, transferable procedural knowledge from diverse sources, unlike factual recall which depends on narrow, document-specific memorization of target facts.
StructTuning achieves 50% of full-corpus performance using only 0.3% of training data by organizing chunks into auto-generated domain taxonomies. The model learns knowledge position within conceptual structures rather than raw text patterns, matching how students learn from textbooks.
AI systems that learn exclusively from data produce uninterpretable representations, inherit statistical biases uncorrected by normative rules, and fail to generalize beyond training distributions. Structured knowledge injection at minimal corpus cost substantially improves performance.
Knowledge Graph of Thoughts (KGoT) achieves 29% improvement on GAIA Level 3 tasks using GPT-4o mini by externalizing reasoning into iteratively constructed KG triples. The approach improves transparency, reduces bias, and enables quality control over reasoning steps.