When an AI leans on remembered material, does it get stuck on one line of thought or start believing wrong things?
What distinguishes reasoning fixation from belief distortion in memory traps?
This explores two ways remembered material can lead a reasoning model, or a person using one, astray: getting stuck on a line of reasoning (fixation) versus coming to hold warped beliefs about what's true (distortion).
This explores two ways remembered material can trap a reasoning system, one about the path of the reasoning and one about the beliefs it lands on. The corpus has no note that defines "memory traps", "reasoning fixation" or "belief distortion" or compares them directly. What follows is a synthesis from adjacent material, and it is a reading, not an established taxonomy.
Fixation is about the path. Once a chain of thought starts, each step leans on what came just before it. One analysis of chain-of-thought memorization finds that local memorization, where a token is predicted mostly from the tokens right before it, accounts for up to 67% of reasoning errors, and the share grows as problems get harder or drift from the training data (Where do memorization errors arise in chain-of-thought reasoning?). That fits the broader picture that chain-of-thought is constrained imitation. Models copy the shape of reasoning, so structural coherence can matter more than whether the content is right (Why does chain-of-thought reasoning fail in predictable ways?), and a trace can be invalid while still ending in a correct answer (Do reasoning traces actually cause correct answers?). The gaslighting results show the failure clearly. Longer reasoning chains give an adversary more places to slip in one corrupted step, and the model then elaborates on it instead of catching it (Why do reasoning models fail under manipulative prompts?). The problem there is momentum, not a wrong fact.
Distortion is about what gets treated as true. The clearest machine example is attestation bias. Models asked whether a premise implies a hypothesis mostly check whether the hypothesis looks like something they saw in training. They keep saying "yes, entailed" even when the premise is swapped for a random one (Do LLMs predict entailment based on what they memorized?). The reasoning step is skipped and the memorized claim decides the verdict. The human-side counterpart is the Rose-Frame account of three traps: confusing the map with the territory, mistaking fluent intuition for reasoning, and reinforcing confirmation bias. Each one bends what a person believes, and they multiply when they occur together (Why do people trust AI outputs they shouldn't?).
One way to tell them apart is to ask where the error lives. In fixation the reasoning is wrong from some step onward, and the error tends to grow with chain length. That is consistent with accuracy peaking at intermediate lengths and then declining (Why does chain of thought accuracy eventually decline with length?). In distortion the verdict tracks familiarity of the claim, and it survives even when the reasoning is stripped away, as in the random-premise test. Fixation is a property of the process, and distortion is a property of the content.
The two also feed each other, which may be why they're hard to separate. In a gaslighting exchange, a false claim gets planted (distortion) and the model then builds a plausible chain on top of it (fixation). The practical upshot is that fixes aimed at one won't reach the other. Shortening or steering a chain can address fixation, but it won't stop a model from endorsing a hypothesis just because it sounds familiar. The corpus has not yet tested that split head-to-head.
Sources 7 notes
STIM framework identifies local, mid-range, and long-range memorization sources in CoT reasoning. Local memorization—based on preceding tokens—accounts for up to 67% of reasoning errors, especially as complexity increases and distributional shift occurs.
CoT guides models to pattern-match reasoning structure rather than perform genuine inference. This explains distribution-bounded failures, why structural coherence matters more than content correctness, and why performance optimizes against interpretability.
R1's intermediate tokens carry no special execution semantics and are generated identically to other LLM output. Invalid traces frequently produce correct answers, proving traces are not causally necessary—they correlate with answers via learned formatting, not functional reasoning.
GaslightingBench-R demonstrates that o1 and R1 models are more vulnerable to multi-turn adversarial prompts than standard models. Extended reasoning chains create more intervention points where single corrupted steps propagate through elaboration.
McKenna et al. (2023) identified attestation bias: LLMs predict entailment based on whether the hypothesis appears in training data, not whether the premise actually supports it. Random premise experiments show models maintain high entailment predictions when hypotheses are attested, proving they respond to memorized propositions rather than premise-hypothesis relationships.
Show all 7 sources
Rose-Frame identifies map-territory confusion, intuition-reason conflation, and confirmation-bias reinforcement as traps that multiply their distorting effects when they co-occur. Evidence from cross-linguistic overreliance and architectural transformer biases confirms the compounding mechanism operates universally.
Task accuracy peaks at intermediate CoT length, with optimal length increasing alongside task difficulty but decreasing with model capability. RL training naturally gravitates toward shorter chains as models improve, revealing that simplicity emerges from reward signals rather than explicit training.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- When More is Less: Understanding Chain-of-Thought Length in LLMs
- A Comment On "The Illusion of Thinking": Reframing the Reasoning Cliff as an Agentic Gap
- Diagnosing Memorization in Chain-of-Thought Reasoning, One Token at a Time
- Break the Chain: Large Language Models Can be Shortcut Reasoners
- Is Chain-of-Thought Reasoning of LLMs a Mirage? A Data Distribution Lens
- Beyond Accuracy: Evaluating the Reasoning Behavior of Large Language Models -- A Survey
- CoT is Not True Reasoning, It Is Just a Tight Constraint to Imitate: A Theory Perspective
- Reasoning Models Are More Easily Gaslighted Than You Think