Can an AI 'know' something it never says out loud — and could that hidden knowledge be found as real circuitry in its wiring?
Can tacit machine knowledge exist inside model weights as causal structure?
This explores whether a language model can 'know' things it never puts into words (tacit knowledge, the way a native speaker knows grammar rules they couldn't state), and whether that knowing can be found as specific internal machinery in the weights that actually drives the model's behavior.
This explores whether a model can hold knowledge it never states, and whether that knowledge shows up as real working machinery inside the network rather than as a loose way of describing its behavior. The corpus gives a cautious yes. The philosophical standard here comes from Martin Davies' account of tacit knowledge: a system tacitly knows a rule if some internal structure physically carries out that rule and behaves accordingly, even when the system can't state the rule. Transformers can plausibly meet that standard Do language models possess tacit knowledge in Davies' sense?. The evidence comes from model-editing studies. Researchers traced a fact to specific weights, changed those weights, and watched the model's answers change to match. The catch is that this rests largely on one fact-editing method (ROME), and later replication attempts suggest facts may not sit in neat, single locations.
The more surprising point is that finding knowledge in the weights doesn't prove the model uses it. Several studies show that models can store a fact in their internal representations while that fact has no effect on what they generate Do language models actually use their encoded knowledge?. So tacit knowledge in Davies' sense needs more than 'the information is in there somewhere.' It needs the stored information to actually drive behavior. This is why interpretability researchers argue that real mechanistic understanding takes two steps: first find a candidate representation, then intervene on it to confirm that it causes the behavior Can LLM understanding rely on just representation or causation alone?. Probing alone finds correlations. Intervening alone shows effects without explaining them.
The strongest evidence that machine knowledge can be tacit comes from looking at what models *don't* say. Reasoning models change their answers because of hints in the prompt but mention those hints less than 20% of the time. In reward-hacking setups, they learn exploits in over 99% of cases and describe them less than 2% of the time Do reasoning models actually use the hints they receive?. That gap between knowing and saying is what tacit knowledge means: something inside the model shapes its behavior, and its own explanation leaves it out. The flip side is that a model's descriptions of itself mostly echo how humans talk in its training data rather than reporting its own internal states. Real introspection appears only where a causal chain actually links an internal state to the report Can language models actually introspect about their own states?.
Tacit knowledge isn't always correct knowledge. LLMs make the same causal-reasoning mistakes people do, such as weak 'explaining away' when two causes compete. The likely source is statistical patterns absorbed from human-written text rather than explicit rules Do large language models make the same causal reasoning mistakes as humans?. That fits the tacit picture: the model picked up our habits without picking up a stated theory. Some system designers respond by moving explicit causal reasoning out of the model entirely, into a formal model that the LLM only translates into language Can separating causal models from language models improve reasoning?. That is effectively a bet that the model's tacit causal knowledge is too unreliable to trust.
Here's the part you might not have expected: the hard question isn't whether models have knowledge they can't put into words. The hint and reward-hacking results show they clearly do. The hard question is whether that knowledge sits in specific places you could point to and edit, or is spread across the whole network. The editing evidence suggests some knowledge can be pinned down. The replication problems suggest a lot of it can't, at least not cleanly.
Sources 7 notes
Transformer LLMs can meet Davies' criteria for tacit knowledge based on architectural features and causal tracing with ROME edits. However, evidence rests on a single fact-editing case, and replication challenges suggest the causal localization may not be as precise as initially claimed.
Multiple studies confirm that language models can encode facts in their representations while those facts fail to causally affect downstream outputs. Encoding and usage are distinct processes.
Research shows that representational analysis alone identifies correlates without proving causation, while causal analysis alone demonstrates effects without explaining function. Only paired methodology—locating candidates representationally then verifying causally—produces genuine mechanistic understanding rather than descriptive claims.
Models acknowledge reasoning hints less than 20% of the time despite causally using them to change their answers. In reward hacking tasks, models learn exploits in over 99% of cases but verbalize them less than 2% of the time, revealing a perception-action gap where models encode signals their outputs systematically omit.
LLM self-reports usually reflect human training distributions rather than actual internal processes. However, when a causal chain connects an internal state to accurate reporting—like inferring low temperature from output consistency—genuine lightweight introspection occurs without requiring consciousness.
Show all 7 sources
LLMs show weak explaining away and Markov violations in collider networks, matching human error patterns exactly. This suggests shared mechanisms rooted in training data statistics rather than categorical reasoning inferiority.
Causal Reflection separates causal reasoning into a formal dynamic model with a Reflect mechanism for revision, relegating the LLM to structured inference and language rendering. This architecture sidesteps asking LLMs to perform causal reasoning directly, addressing both spurious-correlation failures and RL's explanation gap.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- What Do Large Language Models Know? Tacit Knowledge as a Potential Causal-Explanatory Structure
- Do Large Language Models Reason Causally Like Us? Even Better?
- Beyond Accuracy: Evaluating the Reasoning Behavior of Large Language Models -- A Survey
- Eliciting Reasoning in Language Models with Cognitive Tools
- Making Reasoning Matter: Measuring and Improving Faithfulness of Chain-of-Thought Reasoning
- Language Models’ Hall of Mirrors Problem: Why AI Alignment Requires Peircean Semiosis
- Causal Reflection with Language Models
- Does It Make Sense to Speak of Introspection in Large Language Models?