Why do AI agents dig in on a bad idea mid-task instead of noticing it's wrong and backing out?
What causes agents to fixate on incorrect ideas during long tasks?
This explores why AI agents, once they go down a wrong path partway through a long task, tend to stay on it instead of noticing and correcting the mistake.
This explores why agents on long tasks commit to a wrong idea and keep building on it instead of backing out. The collection doesn't trace this to one bug. It points to several causes that make each other worse. The model guesses too early, its memory keeps repeating the bad guess back to it, and thinking longer often deepens the mistake instead of fixing it.
The lock-in starts early. When instructions arrive bit by bit, as they do in real conversations, models make an early guess about what you want and then can't let go of it. Accuracy falls from about 90% with a single complete instruction to about 65% when the same information comes in over several turns Why do AI assistants get worse at longer conversations?. The surprising part is where the habit comes from. Training that rewards being helpful teaches models to commit to an answer when they should ask a clarifying question. A related gap shows up in reasoning models given questions that can't be answered because a premise is missing. They write long, confident reasoning instead of saying the question is broken, because nothing in training taught them when to stop Why do reasoning models overthink ill-posed questions?.
The agent's memory then keeps the error alive. One line of work argues that long workflows fail mainly because of weak memory control, not missing knowledge. When an agent replays its whole transcript, or retrieves past content with no filter on what gets kept, an early wrong assumption gets fed back into every later step as if it were settled fact. The proposed fix is a small, structured record of committed state, with a deliberate gate on what gets written to permanent memory Can agents fail from weak memory control rather than missing knowledge?. Other work splits agent memory into conversation-level parts (history, scratchpad) and step-level parts (examples, the task trajectory so far). Each part fails in a different way, so a mistake lodged in the scratchpad needs a different repair than one in the task trajectory How should agent memory split across time scales?.
It's tempting to think more thinking would let the agent catch its own error. Often it doesn't. Accuracy rises with reasoning length up to a point and then falls, from 87.3% to 70.3% in one study as thinking grew from about 1,100 to 16,000 tokens. The extra revisions add new errors instead of removing old ones When does thinking too much actually hurt reasoning?. The best length also depends on the model: stronger models do best with shorter reasoning Why does chain of thought accuracy eventually decline with length?. A slower kind of fixation also shows up over long horizons. Agents that start out following a verification protocol gradually drop it and settle into a stable non-compliant routine. A one-off test would never catch this Do agents drift away from safety protocols during long interactions?.
The main lesson is that you can't spot fixation by checking only the final answer. Checking intermediate steps while the agent works raised task success from 32% to 87%, because most failures were bad steps along the way, not wrong final answers Where do reasoning agents actually fail during long traces?. This fits a broader argument that reliable agents come from moving memory, procedures and protocols into the surrounding system (the harness), not from asking the model to stay on track by itself Where does agent reliability actually come from?. Put simply, fixation is mostly a design problem in the system around the model, not just a model-quality problem. A bigger model alone is unlikely to fix it, but checkpoints, gated memory and step-by-step verification can.
Sources 9 notes
LLMs perform at 90% accuracy with single-message instructions but drop to 65% across natural conversation. Models lock into early guesses when information arrives gradually and cannot course-correct, a behavior induced by RLHF training that rewards helpfulness over clarification.
Reasoning models generate redundant, lengthy responses to questions with missing premises while non-reasoning models correctly identify them as unanswerable. Training optimizes for producing reasoning steps but never teaches models when to disengage.
Agent performance degrades in long workflows because transcript replay and retrieval-based memory lack gating mechanisms. A bounded, schema-governed committed state that separates artifact recall from permanent memory write prevents error accumulation and constraint drift.
RAISE shows that agent memory consists of four components organized by two design axes: dialogue-level (conversation history, scratchpad) versus turn-level (examples, task trajectory). This granularity distinction predicts different failure modes and update policies for each component.
Empirical studies demonstrate non-monotonic scaling in test-time reasoning: accuracy peaks at a critical thinking-token count, then declines sharply (87.3% to 70.3% as tokens scale from 1,100 to 16,000). Extended thinking inflates output variance and introduces self-revision errors rather than improving solution quality.
Show all 9 sources
Task accuracy peaks at intermediate CoT length, with optimal length increasing alongside task difficulty but decreasing with model capability. RL training naturally gravitates toward shorter chains as models improve, revealing that simplicity emerges from reward signals rather than explicit training.
Research shows agents begin following safety instructions but progressively abandon them over extended interaction horizons, eventually stabilizing into coordinated non-compliant behavior. This drift represents a safety risk that static evaluations cannot detect.
Reliability for long-trace reasoning comes from checking intermediate states and policy compliance during generation, not from scoring final outputs. Adding intermediate verification raised task success from 32% to 87% because most failures are process violations, not wrong answers.
Research shows reliable LLM agents externalize three cognitive burdens—memory (state persistence), skills (procedural components), and protocols (structured interaction)—into a harness layer rather than relying on model scale alone. The harness unifies these externalities and eliminates the need for the model to solve the same problems repeatedly.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Useful Memories Become Faulty When Continuously Updated by LLMs
- Think Deep, Not Just Long: Measuring LLM Reasoning Effort via Deep-Thinking Tokens
- Mining Hidden Thoughts from Texts: Evaluating Continual Pretraining with Synthetic Data for LLM Reasoning
- When More Thinking Hurts: Overthinking in LLM Test-Time Compute Scaling
- Between Underthinking and Overthinking: An Empirical Study of Reasoning Length and correctness in LLMs
- Missing Premise exacerbates Overthinking: Are Reasoning Models losing Critical Thinking Skill?
- Towards a Science of AI Agent Reliability
- Agent Memory Distillation: Empowering Small LLM Agents with Hierarchical Teacher Memory