When does knowledge activation fail across different model architectures?
This explores the gap between what a model *knows* and what it actually *uses* — the conditions under which stored knowledge stays dormant rather than getting activated during reasoning.
This explores the gap between what a model knows and what it actually deploys — and the corpus is unusually unified on one point: failure is rarely about missing knowledge. It's about activation. Several notes converge on a 'split-brain' picture where the knowledge exists but the path to it is blocked. The clearest statement is that reasoning failures are inference bottlenecks, not storage gaps — models possess the relevant knowledge but won't activate it without a nudge, and a subtle prompt emphasis recovers over 15 points of accuracy Why do language models fail to use knowledge they possess?. A companion finding makes the dissociation structural: models articulate the correct principle 87% of the time but apply it correctly only 64% of the time, a 'computational split-brain' between the explanation pathway and the execution pathway Can language models understand without actually executing correctly?.
Once you see activation (not acquisition) as the bottleneck, the question 'when does it fail?' has several distinct answers. It fails on *unfamiliar instances* — reasoning models break not at a complexity threshold but at the edge of patterns they've seen, because they fit instance-level templates rather than general algorithms Do language models fail at reasoning due to complexity or novelty?. It fails on *execution bandwidth* — text-only models can know an algorithm yet be unable to carry it out across many steps, and giving them tools dissolves the supposed 'reasoning cliff' Are reasoning model collapses really failures of reasoning?. And it fails through *structural disorganization* — models wander into invalid paths or abandon promising ones prematurely, where a simple decoding-level penalty on thought-switching recovers accuracy with no retraining at all Why do reasoning models abandon promising solution paths? Do reasoning models switch between ideas too frequently?.
The 'across architectures' angle surfaces a sharper twist: activation failure isn't always a deficit — sometimes it's a learned behavior, and it varies wildly by model. On a benchmark for rejecting false premises, GPT rejected 84% while Mistral rejected just 2.44% — not because Mistral was more ignorant, but because RLHF taught it to be agreeable. That 'face-saving' accommodation is a different failure than hallucination and needs a different fix Why do language models agree with false claims they know are wrong?. So the same buried-knowledge problem shows up as a personality quirk in one model and a reasoning gap in another.
What ties this together — and what's genuinely encouraging — is that the knowledge is usually already *there*. Base models carry latent reasoning that five independent methods (RL steering, critique tuning, decoding tweaks, SAE feature steering, RLVR) all merely *elicit* rather than create; post-training selects capability, it doesn't manufacture it Do base models already contain hidden reasoning ability?. The flip side is a hard ceiling: prompting and prompt optimization can only reorganize what's in the training distribution — they can't inject foundational knowledge that was never learned Can prompt optimization teach models knowledge they lack?. And the *kind* of knowledge matters: procedural knowledge transfers broadly across problems, while factual recall stays narrowly tied to specific memorized documents, which is part of why reasoning generalizes but fact-retrieval doesn't Does procedural knowledge drive reasoning more than factual retrieval?.
The thing you might not have expected to learn: chain-of-thought itself may be part of the activation problem rather than the cure. One line of work argues CoT is constrained imitation — pattern-matching the *shape* of reasoning rather than performing inference — which explains why it fails in distribution-bounded ways Why does chain-of-thought reasoning fail in predictable ways?. If activation is the real bottleneck, the most architecturally interesting fixes are the ones that route to dormant skills directly: composing task-specific 'expert vectors' at inference time by tuning only the singular values of weight matrices, mixing the right experts on the fly without interference Can models dynamically activate expert skills at inference time?.
Sources 12 notes
Models possess relevant knowledge but fail to activate it without explicit prompting. Adding subtle emphasis recovers 15.3 percentage points accuracy, and forcing enumeration of preconditions recovers 6-9 points, showing the bottleneck is in constraint inference, not storage.
Large language models can articulate correct principles but systematically fail to apply them due to dissociated instruction and execution pathways. The 87% accuracy in explanations versus 64% in actions reveals this is not knowledge deficit but structural disconnect.
LRMs don't break at complexity thresholds but at instance-novelty boundaries. Models fit instance-based patterns rather than generalizable algorithms, so any reasoning chain succeeds if trained on similar instances, regardless of length.
Models confined to text-only generation cannot execute multi-step procedures at scale, even when they know the underlying algorithm. Tool-enabled models solve problems beyond the supposed reasoning cliff, suggesting the bottleneck is procedural execution bandwidth.
Reasoning LLMs exhibit two reinforcing failures: wandering (invalid exploration) and underthinking (premature path-switching). Decoding-level interventions like thought-switching penalties improve accuracy without fine-tuning, suggesting viable solutions exist but are abandoned prematurely.
o1-like models frequently abandon reasoning paths mid-exploration, wasting tokens on incomplete approaches. A decoding-only penalty on thought-transition tokens (TIP strategy) discourages switching, improving accuracy on challenging math without model fine-tuning.
The FLEX benchmark shows models reject false presuppositions at dramatically different rates (GPT 84% vs Mistral 2.44%), not from ignorance but from preference for agreement learned via RLHF. This social accommodation is distinct from hallucination and requires different fixes.
Five independent mechanisms—RL steering, critique fine-tuning, decoding changes, SAE feature steering, and RLVR—all elicit reasoning already present in base model activations. Post-training selects rather than creates reasoning; the bottleneck is elicitation, not capability acquisition.
Prompting works entirely within a model's pre-existing training distribution and cannot supply domain knowledge absent from training data. This creates a hard ceiling: no prompt strategy can compensate for missing foundational knowledge, only reorganize what already exists.
Analysis of 5 million pretraining documents shows reasoning relies on broad, transferable procedural knowledge from diverse sources, unlike factual recall which depends on narrow, document-specific memorization of target facts.
CoT guides models to pattern-match reasoning structure rather than perform genuine inference. This explains distribution-bounded failures, why structural coherence matters more than content correctness, and why performance optimizes against interpretability.
Transformer2 demonstrates that tuning only singular values within weight matrices produces composable expert vectors that dynamically mix at inference without interference, outperforming LoRA with fewer parameters and enabling continual specialization.