Giving an AI the right evidence doesn't mean it uses it — it may just be answering from memory.
How does evidence retrieval affect compositional reasoning in language models?
This explores what happens to a model's ability to chain evidence together (compositional reasoning) when you feed it retrieved premises, documents, or context — and the corpus suggests retrieval helps far less than you'd hope, because models lean on memorized associations rather than actually composing the evidence you hand them.
This reads the question as: if you give a language model the right evidence, does it actually reason over that evidence to reach compound conclusions? The corpus's uncomfortable answer is that retrieval and reasoning are only loosely coupled — supplying the correct premises does not guarantee the model uses them. The clearest demonstration is attestation bias: models predict that a premise entails a hypothesis based on whether the hypothesis looks familiar from training, not on whether the retrieved premise actually supports it. Swap in a random, irrelevant premise and the model still says 'entailed' as long as the conclusion is memorable Do LLMs predict entailment based on what they memorized?. Retrieval, in other words, can be decorative — the evidence is present but not load-bearing.
This connects to a deeper finding about *why* evidence gets ignored: when retrieved context conflicts with strong parametric priors, the priors win. Models generate outputs inconsistent with the very documents in their context window because training-time associations dominate, and no amount of prompting ('use only the provided text') reliably overrides them — you need causal intervention in the internal representations to force it Why do language models ignore information in their context?. So the first thing retrieval does to compositional reasoning is expose a competition the model often loses: memorized knowledge versus supplied evidence.
There's also a purely mechanical cost. Retrieval means longer inputs, and reasoning accuracy degrades sharply as inputs grow — dropping from 92% to 68% with just a few thousand tokens of padding, well below the context-window limit, and chain-of-thought doesn't rescue it Does reasoning ability actually degrade with longer inputs?. The more evidence you stuff in, the worse the multi-step reasoning gets, independent of whether the evidence is relevant. This is a quiet tax that undercuts the whole premise of 'just retrieve more.'
The reason retrieval can't paper over this is structural: what looks like compositional reasoning in transformers is largely memorized subgraph matching. Models succeed on in-distribution compositions by recalling computation patterns seen in training, and fail badly on genuinely novel combinations, with errors compounding across steps Do transformers actually learn systematic compositional reasoning?. Retrieval can only help if the composition the evidence implies is already close to something the model has seen — and relatedly, models reason semantically, through token associations, rather than symbolically; when you decouple meaning from the logical form, performance collapses even with the correct rules sitting in context Do large language models reason symbolically or semantically?. Chain-of-thought over retrieved evidence often just reproduces familiar reasoning *shapes* rather than performing new inference Does chain-of-thought reasoning reveal genuine inference or pattern matching?.
The lateral takeaway worth carrying away: the field frames retrieval failures as a retrieval problem (get better documents) — but this corpus reframes it as a *composition* problem. Even perfect evidence retrieval runs into a model that prefers attested conclusions, defers to its priors, degrades under length, and matches memorized subgraphs instead of composing fresh ones. And there's a hint of a fix in the same territory: models actually rank tokens by functional importance internally, preferentially preserving symbolic-computation steps Which tokens in reasoning chains actually matter most? — suggesting the machinery to weight real evidence over surface familiarity exists inside the model, it just isn't reliably in charge.
Sources 7 notes
McKenna et al. (2023) identified attestation bias: LLMs predict entailment based on whether the hypothesis appears in training data, not whether the premise actually supports it. Random premise experiments show models maintain high entailment predictions when hypotheses are attested, proving they respond to memorized propositions rather than premise-hypothesis relationships.
Research demonstrates that LMs generate outputs inconsistent with their context because parametric knowledge from training dominates over in-context information. Textual prompting alone cannot override strong priors; causal intervention in representations is required.
FLenQA shows reasoning accuracy drops from 92% to 68% at just 3000 tokens of padding, far below context window capacity. The degradation is task-agnostic, uncorrelated with language modeling performance, and persists even with chain-of-thought prompting.
Research shows transformers succeed on in-distribution tasks by memorizing computation subgraphs from training data, not by learning systematic rules. They fail drastically on novel compositions, with errors compounding across reasoning steps.
When semantic content is decoupled from reasoning tasks, LLM performance collapses even with correct rules in context. Models rely on parametric commonsense and token associations rather than formal logical manipulation, constraining reasoning to training distribution semantics.
Show all 7 sources
CoT works by constraining models to reproduce familiar reasoning patterns from training, not by enabling novel symbolic reasoning. Performance degrades predictably under distribution shifts—the signature of imitation rather than capability emergence.
Greedy likelihood-preserving pruning reveals six functional token categories; symbolic computation tokens are preferentially preserved while grammar and meta-discourse are pruned first. Student models trained on these pruned chains outperform those trained on frontier-model compression.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Beyond Accuracy: Evaluating the Reasoning Behavior of Large Language Models -- A Survey
- Is Chain-of-Thought Reasoning of LLMs a Mirage? A Data Distribution Lens
- When More is Less: Understanding Chain-of-Thought Length in LLMs
- Language models show human-like content effects on reasoning tasks
- On the Reasoning Capacity of AI Models and How to Quantify It
- Simple Linguistic Inferences of Large Language Models (LLMs): Blind Spots and Blinds
- Diagnosing Memorization in Chain-of-Thought Reasoning, One Token at a Time
- Do LLMs Encode Functional Importance of Reasoning Tokens?