INQUIRING LINE

Why does a bad call early in a long AI task often stay hidden until much later, or forever?

What causes delays between wrong decisions and visible consequences in long tasks?

This explores why a bad choice early in a long task, whether an agent's or a reasoning model's, often doesn't show up until much later or at all, and what creates that gap.


This explores why a bad choice early in a long task often stays invisible until much later, or forever. The corpus has no note on the delay itself. But several notes describe the ingredients: mistakes that hide inside the process, small errors that pile up, and evidence that only exists after the agent has acted.

The first ingredient is that we mostly look at the end. Where do reasoning agents actually fail during long traces? argues that most failures in long traces are process violations, not wrong answers. Checking intermediate states raised task success from 32% to 87%. The slip happens at step 3, but the only thing anyone inspects is step 30, and sometimes the final answer looks fine even though the process broke a rule along the way. What three separate factors drive chain-of-thought performance? adds a second reason the slip stays quiet. Genuine reasoning is happening, but it accumulates error with each step, so no single step looks alarming. Length makes this worse. When does thinking too much actually hurt reasoning? found accuracy falling from 87.3% to 70.3% as thinking grew from 1,100 to 16,000 tokens, partly through self-revision errors. A longer run gives an early slip more room to be built on and buried.

The second ingredient is that the world doesn't always answer right away. When should an agent stop acting and admit failure? studied 28,000+ tasks. It found that a task's infeasibility often emerges only after interaction, so at the moment of the bad decision there is nothing to see yet. Agents then abstain too late or never. What they lack is the timing, not the ability to stop, and the note reports this can be fixed at inference time without retraining.

The third ingredient is that a wrong step can feel settled. When should an agent actually stop and deliberate? triggers extra deliberation only when sampled actions disagree. Read the other way, a mistake the model makes consistently looks like a sure thing and draws no scrutiny. The run's length gives no warning either. Does longer reasoning actually mean harder problems? shows that trace length tracks how close a problem is to training data, not how hard it is. A long, fluent trace doesn't tell you it's on track.

The notes point to one way to shrink the gap, which is to tighten the feedback loop. What predicts success in ultra-long-horizon agent tasks? found that repeated benchmark-edit-incorporate cycles predicted success across 17 frontier models better than initial quality. Most models quit early or burned budget without learning from feedback. Can confidence patterns reveal overthinking versus underthinking? hints at a complement. Confidence patterns can be read during a run to catch reasoning that is looping or under-exploring, with no retraining. Together they suggest that consequences arrive late because we wait for them. Checking and measuring midway pulls them forward.


Sources 8 notes

Where do reasoning agents actually fail during long traces?

Reliability for long-trace reasoning comes from checking intermediate states and policy compliance during generation, not from scoring final outputs. Adding intermediate verification raised task success from 32% to 87% because most failures are process violations, not wrong answers.

What three separate factors drive chain-of-thought performance?

A shift cipher study decomposed CoT into three independent factors: output probability alone swings accuracy from 26% to 70%, memorization matches pre-training frequency patterns, and genuine reasoning exists but accumulates error with each step. This resolves the reason-or-memorize debate by showing LLMs do both simultaneously.

When does thinking too much actually hurt reasoning?

Empirical studies demonstrate non-monotonic scaling in test-time reasoning: accuracy peaks at a critical thinking-token count, then declines sharply (87.3% to 70.3% as tokens scale from 1,100 to 16,000). Extended thinking inflates output variance and introduces self-revision errors rather than improving solution quality.

When should an agent stop acting and admit failure?

Across 28,000+ tasks, LLM agents either abstain too late or never, even though infeasibility often only emerges after interaction. The missing competence is knowing *when* to stop, not *whether* they can—and this can be fixed at inference time without retraining.

When should an agent actually stop and deliberate?

SAND uses self-consistency sampling to flag uncertainty: if N policy samples all match the expert action, skip deliberation; if they diverge, trigger execution-guided critiques. This step-level compute allocation lets agents deliberate only at genuinely uncertain decision points.

Show all 8 sources
Does longer reasoning actually mean harder problems?

Controlled A* maze experiments show trace length correlates with difficulty only in-distribution but decouples entirely out-of-distribution. Trace length primarily reflects recall of training schemas, not adaptive computation.

What predicts success in ultra-long-horizon agent tasks?

Across 17 frontier models on 36 expert-curated optimization tasks, repeated benchmark-edit-incorporate cycles within a wall-clock budget proved the dominant success predictor. Most models terminated early or burned budget unproductively; Claude Opus 4.6 stood out as persistent.

Can confidence patterns reveal overthinking versus underthinking?

ReBalance uses confidence variance and overconfidence as diagnostic signals to apply training-free steering vectors that reduce overthinking redundancy while promoting exploration during underthinking, improving accuracy across models from 0.5B to 32B parameters.

Papers this line draws on 8

The research behind the notes this line reads — ranked by how closely each paper relates.