SYNTHESIS NOTE
Topics›Training Fine Tuning›this note

Does procedural knowledge drive reasoning more than factual retrieval?

Explores whether models learn reasoning through general procedures across diverse documents rather than memorizing specific facts. This matters for understanding what pretraining data actually teaches models to reason.

Synthesis note · 2026-02-22 · sourced from Training Fine Tuning
What kind of thing is an LLM really? How do you build domain expertise into general AI models? How do you navigate synthesis across fragmented research topics?

The "Procedural Knowledge in Pretraining Drives Reasoning" paper analyzes which pretraining documents most influence LLM reasoning by ranking 5 million documents by their influence on model completions. The finding: the approach to reasoning that models use is unlike retrieval. For reasoning tasks, positively influential documents contain procedural knowledge — descriptions of how to get to a solution — rather than the specific facts needed for the answer.

Three contrasts with factual recall:

  1. Generality: models rely on a broader, more general set of documents when reasoning than when answering factual questions. Factual recall draws on a narrow set of documents containing the target fact. Reasoning draws on a diffuse set of documents performing similar procedures.

  2. Transferability: documents have similar influence on reasoning queries that require applying the same procedure to different numbers. The procedural knowledge transfers across specific instances — it's the method, not the content, that the model has learned.

  3. Reliance distribution: the model needs to see factual information more often (across more documents) to memorize it, while procedural patterns can be learned from fewer but more diverse demonstrations.

This connects to the knowledge/reasoning layer separation. Since Why does reasoning training help math but hurt medical tasks?, the procedural knowledge finding provides the data-level explanation for the architectural finding: lower layers store memorized facts (requiring document-specific exposure), while higher layers encode procedural strategies (learnable from general demonstrations).

The implication for training data curation: reasoning capability benefits more from diverse demonstrations of procedures than from exhaustive factual coverage. Quality and diversity of reasoning demonstrations may matter more than volume for building reasoning capability — consistent with Can models improve themselves on tasks without verifiable answers?.

Inquiring lines that read this note 160

This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.

How do professional roles and expertise transform with AI-generated content? Can AI-generated outputs constitute genuine knowledge or valid claims? Can prompting inject entirely new knowledge into language models? How do neural networks separate factual knowledge from reasoning abilities? How faithfully do LLMs reflect their actual reasoning in outputs and explanations? How do training data properties shape reasoning capability development? Why do reasoning models fail at systematic problem-solving and search? How should iterative research systems allocate reasoning per search step? What capability tradeoffs emerge when scaling model reasoning abilities? Does decoupling planning from execution improve multi-step reasoning accuracy? How do knowledge graphs enable efficient multi-hop reasoning over alternatives? Do base models contain latent reasoning that training can unlock? How do LLMs distinguish causal reasoning from temporal and semantic associations? How does latent reasoning compare to verbalized chain-of-thought? Do reasoning traces faithfully represent or merely mimic actual model reasoning? Why does training format shape reasoning strategy more than domain content? How can identical external performance mask different internal representations? How does AI adoption affect human skill development and labor equality? What memory architectures best support persistent reasoning across extended interactions? Why does finetuning cause catastrophic forgetting of model capabilities? Why do LLM chatbots fail as independent therapeutic agents? How do transformer attention mechanisms implement memory and algorithmic functions? Why do continual learning scenarios trigger catastrophic forgetting and interference? What factors beyond surface content determine how readers extract meaning differently? Does fine-tuning modify underlying model capabilities or only behavioral outputs? What makes weaker teacher models effective for stronger student training? How does memorization interact with learning and generalization? Can alternative training methods improve on supervised fine-tuning for language models? Why do semantic similarity and task relevance diverge in vector embeddings? How can models identify insufficient information and respond appropriately without guessing? Do language models perform faithful symbolic reasoning independent of semantic grounding? When does architectural design matter more than raw model capacity? How does reasoning graph topology affect breakthrough insights and generalization? Which computational strategies best support reasoning in language models? What pretraining choices and baseline capability constrain reinforcement learning gains? How do training priors constrain what context information can override? What limits mechanistic interpretability's ability to characterize models? Why does verification consistently lag behind AI generation? Why does supervised fine-tuning improve accuracy while degrading reasoning quality? Does self-reflection enable models to reliably correct their errors? Does reinforcement learning teach reasoning or just when to reason? How can AI agents autonomously learn and transfer skills across tasks? What determines success in training models on multiple tasks? How effectively do deterministic tools improve language model reasoning on formal tasks? Can prompting strategies overcome LLM biases without model fine-tuning? Do language models develop causal world models or rely on statistical patterns?

Related concepts in this collection 5

This note in its neighbourhood — explore the map, then jump to a related concept in the list below.

Concept map
16 direct connections · 164 in 2-hop network ·dense cluster Open in graph ↗
Does procedural knowledge drive reasoning more t… Why does reasoning training help math but hurt med… Can models improve themselves on tasks without ver… Do base models already contain hidden reasoning ab… Do foundation models learn world models or task-sp… Can text-trained models compress images better tha…

Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph

your link semantically near linked from elsewhere

Related papers in this collection 8

Papers most semantically related to this note, ranked by cosine similarity in the embedding space.

Search by related questions 4

Suggested questions this note speaks to — click to search the collection, or type your own.

Why does reasoning transfer across different numbers but factual recall does not?How many document exposures does procedural knowledge versus factual information require?Can smaller amounts of diverse reasoning demonstrations replace exhaustive factual training data?Why do higher network layers capture procedural knowledge but lower layers store facts?
Original note title

procedural knowledge in pretraining documents drives reasoning generalization unlike factual retrieval which requires document-specific memorization