Why does a bad call early in a long AI task often stay hidden until much later, or forever?
What causes delays between wrong decisions and visible consequences in long tasks?
This explores why a bad choice early in a long task, whether an agent's or a reasoning model's, often doesn't show up until much later or at all, and what creates that gap.
This explores why a bad choice early in a long task often stays invisible until much later, or forever. The corpus has no note on the delay itself. But several notes describe the ingredients: mistakes that hide inside the process, small errors that pile up, and evidence that only exists after the agent has acted.
The first ingredient is that we mostly look at the end. Where do reasoning agents actually fail during long traces? argues that most failures in long traces are process violations, not wrong answers. Checking intermediate states raised task success from 32% to 87%. The slip happens at step 3, but the only thing anyone inspects is step 30, and sometimes the final answer looks fine even though the process broke a rule along the way. What three separate factors drive chain-of-thought performance? adds a second reason the slip stays quiet. Genuine reasoning is happening, but it accumulates error with each step, so no single step looks alarming. Length makes this worse. When does thinking too much actually hurt reasoning? found accuracy falling from 87.3% to 70.3% as thinking grew from 1,100 to 16,000 tokens, partly through self-revision errors. A longer run gives an early slip more room to be built on and buried.
The second ingredient is that the world doesn't always answer right away. When should an agent stop acting and admit failure? studied 28,000+ tasks. It found that a task's infeasibility often emerges only after interaction, so at the moment of the bad decision there is nothing to see yet. Agents then abstain too late or never. What they lack is the timing, not the ability to stop, and the note reports this can be fixed at inference time without retraining.
The third ingredient is that a wrong step can feel settled. When should an agent actually stop and deliberate? triggers extra deliberation only when sampled actions disagree. Read the other way, a mistake the model makes consistently looks like a sure thing and draws no scrutiny. The run's length gives no warning either. Does longer reasoning actually mean harder problems? shows that trace length tracks how close a problem is to training data, not how hard it is. A long, fluent trace doesn't tell you it's on track.
The notes point to one way to shrink the gap, which is to tighten the feedback loop. What predicts success in ultra-long-horizon agent tasks? found that repeated benchmark-edit-incorporate cycles predicted success across 17 frontier models better than initial quality. Most models quit early or burned budget without learning from feedback. Can confidence patterns reveal overthinking versus underthinking? hints at a complement. Confidence patterns can be read during a run to catch reasoning that is looping or under-exploring, with no retraining. Together they suggest that consequences arrive late because we wait for them. Checking and measuring midway pulls them forward.
Sources 8 notes
Reliability for long-trace reasoning comes from checking intermediate states and policy compliance during generation, not from scoring final outputs. Adding intermediate verification raised task success from 32% to 87% because most failures are process violations, not wrong answers.
A shift cipher study decomposed CoT into three independent factors: output probability alone swings accuracy from 26% to 70%, memorization matches pre-training frequency patterns, and genuine reasoning exists but accumulates error with each step. This resolves the reason-or-memorize debate by showing LLMs do both simultaneously.
Empirical studies demonstrate non-monotonic scaling in test-time reasoning: accuracy peaks at a critical thinking-token count, then declines sharply (87.3% to 70.3% as tokens scale from 1,100 to 16,000). Extended thinking inflates output variance and introduces self-revision errors rather than improving solution quality.
Across 28,000+ tasks, LLM agents either abstain too late or never, even though infeasibility often only emerges after interaction. The missing competence is knowing *when* to stop, not *whether* they can—and this can be fixed at inference time without retraining.
SAND uses self-consistency sampling to flag uncertainty: if N policy samples all match the expert action, skip deliberation; if they diverge, trigger execution-guided critiques. This step-level compute allocation lets agents deliberate only at genuinely uncertain decision points.
Show all 8 sources
Controlled A* maze experiments show trace length correlates with difficulty only in-distribution but decouples entirely out-of-distribution. Trace length primarily reflects recall of training schemas, not adaptive computation.
Across 17 frontier models on 36 expert-curated optimization tasks, repeated benchmark-edit-incorporate cycles within a wall-clock budget proved the dominant success predictor. Most models terminated early or burned budget unproductively; Claude Opus 4.6 stood out as persistent.
ReBalance uses confidence variance and overconfidence as diagnostic signals to apply training-free steering vectors that reduce overthinking redundancy while promoting exploration during underthinking, improving accuracy across models from 0.5B to 32B parameters.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Think Deep, Not Just Long: Measuring LLM Reasoning Effort via Deep-Thinking Tokens
- Does Thinking More always Help? Understanding Test-Time Scaling in Reasoning Models
- Is Chain-of-Thought Reasoning of LLMs a Mirage? A Data Distribution Lens
- When More is Less: Understanding Chain-of-Thought Length in LLMs
- Agentic Abstention: Do Agents Know When to Stop Instead of Act?
- The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexity
- Diagnosing Harmful Continuation in Answer-Correct Long-CoT Training Traces
- Missing Premise exacerbates Overthinking: Are Reasoning Models losing Critical Thinking Skill?