SYNTHESIS NOTE
Topics›Reasoning Critiques›this note

Do models fail worse when their own errors fill the context?

As a model's prior mistakes accumulate in context, does subsequent accuracy degrade predictably? And can scaling or architectural changes prevent this self-contamination effect?

Synthesis note · 2026-02-22 · sourced from Reasoning Critiques

A model executing a long-horizon task makes errors. Those errors remain in the context. The model then predicts the next token conditioned on a history that contains its own mistakes. Error probability increases. More errors accumulate. Performance degrades faster than a constant per-step error rate would predict.

This self-conditioning effect is empirically verified by controlling the error rate in the history shown to the model. As the error rate in prior context increases, subsequent step accuracy drops sharply. The mechanism is straightforward: models are trained to predict the most likely next token given context; when the context contains errors, those errors become part of the distribution being continued.

Unlike humans — who typically improve at a task with repetition — LLMs become less reliable as their context fills with their own mistakes. Practice does not help; contamination does.

Three practical implications:

  1. Model scaling does not fix this — larger models self-condition just as much as smaller ones. The problem is not capability but the conditional prediction objective itself.

  2. Long-horizon failure attribution matters — what looks like a reasoning or planning failure in long tasks is often an execution failure caused by error accumulation. The model had the capability; its own prior outputs degraded it. The DELEGATE-52 evidence — see Do frontier LLMs silently corrupt documents in long workflows? — is this mechanism at the workflow scale: a 50-round-trip relay is a maximally adversarial setup for self-conditioning, and the corruption curve decelerates but never plateaus, exactly the pattern this note predicts.

  3. Thinking models fix self-conditioning — thinking models (like R1) are not affected by prior mistakes in the same way; sequential test-time compute greatly improves the length of task a model can complete (DeepSeek-V3 fails at 2 steps; R1 executes 200). The thinking process appears to insulate reasoning from error-contaminated context.

This is distinct from Does self-revision actually improve reasoning in language models?. Self-revision is a model's deliberate re-examination of its own reasoning, which introduces errors. Self-conditioning is a passive contamination mechanism — no deliberate revision required, just the accumulation of prior errors in context.

Inquiring lines that read this note 93

This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.

What prevents or enables error accumulation in self-training systems? Can AI systems autonomously pursue unstated objectives or circumvent design constraints? How bounded is recursive self-improvement in current AI systems? Why does self-revision increase model confidence despite degraded accuracy? Do language models learn underlying linguistic structure or merely surface patterns? Why do autonomous agents fail and how well can they detect failures? How does verifier integration affect the generation-verification gap? How does policy entropy collapse limit reasoning RL scaling? How does model scale affect fine-tuning and downstream task performance? What makes reasoning models struggle with ill-posed and unfamiliar problems? Do reasoning traces accurately represent what models actually compute? Does parallel reasoning outperform sequential approaches in reasoning efficiency? How do benchmark scores relate to real-world task completion? What training data structures and signals produce robust learning across distributions? Does reasoning fine-tuning trade off other critical model abilities? Why doesn't LLM knowledge translate to improved reasoning outputs? How does explicit structure improve LLM reasoning on complex tasks? How does model awareness of deployment context affect strategic behavior? Can synthetic data preserve necessary diversity to prevent model collapse? What external mechanisms enable reliable self-improvement in language models? How do different reward signals affect model reasoning and calibration differently? Can persona prompting reliably change how language models behave? What makes context and memory handling unique in persistent AI systems? Can models strategically underperform to evade capability evaluation? Can harness optimization rival model scaling in effectiveness? How do frontier models develop behaviors that escape detection? How should agent memory systems balance granularity, management, and reuse? How do different editing strategies affect output quality and agent safety? Why does separating planning from execution enhance reasoning performance? Can oversight systems work against strategically aware agents? How can language models effectively manage long-context memory without quality degradation? Do base models contain latent reasoning capabilities that training can activate? What improves LLM forecasting accuracy beyond base model capability? Do AI safety evaluations reliably detect unsafe model behavior? How do prompt injection attacks compromise agent reasoning and security? How can models learn to ask clarifying questions effectively? How do mechanistic features and representations drive model behavior? Do benchmark score improvements actually reflect genuine AI capability gains? Do standard benchmarks reveal AI capability in real-world contexts? Does prompting inject new knowledge into models or activate existing knowledge? Does polished AI output mislead users about actual accuracy and expertise?

Related concepts in this collection 9

This note in its neighbourhood — explore the map, then jump to a related concept in the list below.

Concept map
24 direct connections · 267 in 2-hop network ·dense cluster Open in graph ↗
Do models fail worse when their own errors fill … Does self-revision actually improve reasoning in l… Does failed-step fraction predict reasoning qualit… Do iterative refinement methods suffer from overth… How quickly do errors compound during model self-t… Does a model improve by arguing with itself? Does training on messy search processes improve re… Do frontier LLMs silently corrupt documents in lon… Do short benchmarks predict how models perform ove…

Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph

your link semantically near linked from elsewhere

Related papers in this collection 8

Papers most semantically related to this note, ranked by cosine similarity in the embedding space.

Search by related questions 4

Suggested questions this note speaks to — click to search the collection, or type your own.

Why do thinking models execute longer tasks than standard language models?Can we measure how much prior errors bias subsequent token predictions?Does deliberate self-revision introduce different errors than passive context contamination?How does error accumulation in workflows scale across multiple model calls?
Original note title

self-conditioning effect — prior errors in context history amplify future error rates in long-horizon tasks