Towards Mechanistically Understanding Why Memorized Knowledge Fails to Generalize in Large Language Model Finetuning

Fine-tuning LLMs to inject new knowledge faces a critical challenge: LLMs can quickly memorize new facts, yet fail to use them for downstream reasoning tasks. We formalize this failure as the Knowing–Using Gap, characterized by an accuracy gap and a temporal lag between memorization and generalization. To understand this phenomenon, we fine-tune LLMs with unseen knowledge and monitor the spatial permeation dynamics of the knowledge internally using a novel intervention technique called self-patching. Self-patching identifies activation locations where relocating representations substantially improves failed generalization cases. These results are consistent with a knowledge-circuit misalignment hypothesis: memorized representations can exist internally but may not be routed to computation-effective layers. To demonstrate the practicality of this diagnostic finding, we design a simple heuristic strategy which recovers 58–75% of the oracle headroom in generalization failure. Experiments are done cross-domain for the robustness of this finding. Code and data are open at https://anonymous.4open.science/r/Mem2Gen-71FF.

Introduction. Large language models (LLMs) excel at varieties of tasks but face significant challenges in adapting to unseen information, necessitating effective methods for post-training knowledge updates. While there are approaches like retrieval-augmented generation (RAG) and knowledge editing Meng et al., Gupta et al. [2024], fine-tuning remains a fundamental paradigm for knowledge updating, as it not only operates on parametric memory end-to-end but also injects knowledge in a way that can be reused by the model’s existing reasoning capabilities. Despite sufficient capacity to fit new data Morris et al. [2025], Allen-Zhu and Li, LLMs exhibit a “remembering but not using” failure Ovadia et al. [2024], Soudani et al. [2024], Zhong et al. [2023], Cohen et al. [2024], Berglund et al. as shown in Figure 1: models can memorize new facts (e.g., “Sydney is located in [Entity]”) but fail to reliably use them in downstream reasoning (e.g., “The capital of the country Sydney located is ...”) Zhong et al. [2023], Cohen et al. [2024], Yao et al.

Discussion / Conclusion. We identified the Knowing–Using Gap: fine-tuned LLMs memorize injected facts yet fail to use them in multi-hop reasoning. Using self-patching as a causal diagnostic, we trace this failure to knowledge–circuit misalignment—knowledge is encoded in storage-oriented early or late layers but does not permeate into the mid-layer circuits required for reasoning. The gap is thus a routing problem, not a capacity one—knowledge resides in the model, just not where reasoning happens. It is also partially reversible: a heuristic recovers 58–75% of oracle headroom, and the effect replicates across models, scales, and domains. Together, these results recast fine-tuning’s generalization failure as a tractable alignment problem, and point to alignment-aware training as a principled path forward.