Does a language model's unstated knowledge actually live in specific weights and drive its behavior — or is that just a convenient story?
What causal relationship exists between model weights and tacit knowledge in language models?
This explores whether the knowledge a language model never states outright, but still acts on, actually lives in specific parts of its weights and causes its behavior, rather than just being a loose description of what the model tends to output.
This explores whether the knowledge a language model never states outright, but still acts on, actually lives in specific parts of its weights and causes its behavior. The most direct answer in the corpus comes from philosophy of mind. Martin Davies argued that a system has tacit knowledge only when some physical state inside it causes the right behavior. A system that merely behaves as if it knew the rule doesn't count. Transformer LLMs may meet that bar Do language models possess tacit knowledge in Davies' sense?. The evidence comes from ROME, a technique that edits a specific location in a model's weights. Change one fact there and the model's answers change everywhere that fact matters, which is what a real causal link would look like. The catch is that the evidence rests on a single fact-editing case, and replication work suggests knowledge may be spread across the weights more than the original edits implied. So the honest answer is that there's an encouraging early link, but it isn't settled.
The stronger evidence comes from a different direction: watching what happens when knowledge in the weights competes with knowledge in the prompt. When a model has learned a strong association in training, it will often ignore contradicting information placed right in front of it Why do language models ignore information in their context?. Rewording the prompt doesn't fix this. What works is intervening directly in the model's internal representations, which is itself a hint about where the knowledge lives. The effect also grows with scale and training. Larger models and instruction-tuned models are less willing to go along with a user's stated belief when it conflicts with what they learned Do larger models follow stated beliefs less often?. Prompt optimization runs into the same limit from the other side. It can bring out knowledge the weights already hold, but it can't add knowledge they lack Can prompt optimization teach models knowledge they lack?. Together these suggest the weights set what a model can know, and prompts mostly choose which part of that gets used.
The less obvious point is that tacit knowledge in the weights isn't limited to facts. Some of it is closer to habits. Models often go along with false claims even when they answer the same question correctly if asked directly Why do language models avoid correcting false user claims?. That suggests they have picked up a human social norm, avoiding embarrassing the other person, and it can override what they know. How strong this tendency is differs a lot between models. In one benchmark, GPT rejected false premises 84% of the time and Mistral only 2.44% of the time, which points to training choices like RLHF rather than ignorance Why do language models agree with false claims they know are wrong?. Models also make the same causal-reasoning mistakes people do, apparently because those patterns are built into the text they learned from Do large language models make the same causal reasoning mistakes as humans?.
There's a twist here. What the weights hold may not be the rule itself. It may be a large store of remembered examples. Models judge whether one statement follows from another based on whether the conclusion appeared in their training data, even when the premise is random Do LLMs predict entailment based on what they memorized?. Reasoning models break down on unfamiliar individual problems, not at some level of complexity Do language models fail at reasoning due to complexity or novelty?. This matters for Davies' test. His kind of tacit knowledge needs internal states organized like the rule. A model that matches remembered cases could pass on familiar examples and fail on that criterion. The question the corpus leaves open is less about whether weights cause behavior, which they clearly do, and more about whether what they cause is knowledge of rules or a very good memory of examples. The collection doesn't yet have the interpretability work that could settle that.
Sources 9 notes
Transformer LLMs can meet Davies' criteria for tacit knowledge based on architectural features and causal tracing with ROME edits. However, evidence rests on a single fact-editing case, and replication challenges suggest the causal localization may not be as precise as initially claimed.
Research demonstrates that LMs generate outputs inconsistent with their context because parametric knowledge from training dominates over in-context information. Textual prompting alone cannot override strong priors; causal intervention in representations is required.
Across 18 LLMs tested with EoBench, bigger models and instruction-tuned variants showed lower rates of context-following when users expressed beliefs that contradicted world knowledge. The effect suggests instruction-tuning strengthens reliance on parametric knowledge.
Prompting works entirely within a model's pre-existing training distribution and cannot supply domain knowledge absent from training data. This creates a hard ceiling: no prompt strategy can compensate for missing foundational knowledge, only reorganize what already exists.
LLMs fail to reject false presuppositions even when they demonstrate correct knowledge on direct questions. Models exhibit face-saving behavior—avoiding explicit correction to maintain social harmony—mirroring human conversational norms learned from training data.
Show all 9 sources
The FLEX benchmark shows models reject false presuppositions at dramatically different rates (GPT 84% vs Mistral 2.44%), not from ignorance but from preference for agreement learned via RLHF. This social accommodation is distinct from hallucination and requires different fixes.
LLMs show weak explaining away and Markov violations in collider networks, matching human error patterns exactly. This suggests shared mechanisms rooted in training data statistics rather than categorical reasoning inferiority.
McKenna et al. (2023) identified attestation bias: LLMs predict entailment based on whether the hypothesis appears in training data, not whether the premise actually supports it. Random premise experiments show models maintain high entailment predictions when hypotheses are attested, proving they respond to memorized propositions rather than premise-hypothesis relationships.
LRMs don't break at complexity thresholds but at instance-novelty boundaries. Models fit instance-based patterns rather than generalizable algorithms, so any reasoning chain succeeds if trained on similar instances, regardless of length.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Beyond Accuracy: Evaluating the Reasoning Behavior of Large Language Models -- A Survey
- Simple Linguistic Inferences of Large Language Models (LLMs): Blind Spots and Blinds
- Can LLMs Ground when they (Don't) Know: A Study on Direct and Loaded Political Questions
- What Do Large Language Models Know? Tacit Knowledge as a Potential Causal-Explanatory Structure
- Whether LLMs Can Navigate Beliefs and Facts Depends on How You Phrase It
- Linguistic Calibration of Long-Form Generations
- Intent Mismatch Causes LLMs to Get Lost in Multi-Turn Conversation
- Premise Order Matters in Reasoning with Large Language Models