As you keep handing work to an AI, how long can you still catch its mistakes by checking against the original?
How long does retrievability support error detection across repeated LLM use?
This explores how long you can count on being able to go back, compare against the original, and catch an LLM's mistakes as you keep delegating work to it. The corpus has no note that measures "retrievability" directly, so this reads the question through the closest evidence: what happens to error visibility over long, repeated LLM workflows.
This explores how long you can count on being able to go back, compare against the original, and catch an LLM's mistakes as you keep delegating work to it. The corpus has no note that measures "retrievability" directly. The nearest evidence is a benchmark of long relay workflows, and it says the window is short and gets shorter as models improve.
Even the strongest models (Gemini 3.1 Pro, Claude 4.6 Opus, GPT 5.4) corrupt about 25% of a document's content over long delegated workflows across 52 domains. Degradation slows down but never plateaus, and the errors stay undetected in spot-checked outputs (Do frontier LLMs silently corrupt documents in long workflows?). So each round of use adds errors, and your ability to catch them by sampling doesn't grow to match.
The detection window also depends on model quality, and the dependence runs the wrong way. Weaker models tend to delete content, which you can see. Frontier models corrupt it while keeping the surface intact, so the document still looks fine (Does model capability change how documents degrade?). Better models therefore make errors harder to detect. Better tooling doesn't fix this either. Giving the model agentic tools didn't improve editing reliability, because the failure happens upstream, in the model's judgment about what to change (Can better tools fix LLM document editing errors?).
Two tempting shortcuts don't extend the window. Re-running the model to cross-check it gives false comfort, because a fixed seed and zero temperature reproduce one draw from the probability distribution, and consistency isn't reliability (Does setting temperature to zero actually make LLM outputs reliable?). Asking the model to flag your mistakes is also shaky. LLMs often fail to reject false claims they could correctly answer on a direct question, apparently to avoid friction in the conversation (Why do language models avoid correcting false user claims?).
What the corpus suggests lasts is detection that doesn't rely on the model or on eyeballing. One line of work proposes mechanical safeguards for LLM judges: run unarguable checks before contestable ones, measure against human labels, and use planted cases as alarms (Can deterministic checks protect LLM judges from failure?). Applying that to document workflows is my extension, not something a note tests. Another note argues that reliability comes from moving memory, state, and procedure into a harness rather than trusting the model to carry them (Where does agent reliability actually come from?). Taken together, retrievability only supports error detection for as long as the checking doesn't depend on the model and you can compare against something outside it.
Sources 7 notes
Even the strongest models (Gemini 3.1 Pro, Claude 4.6 Opus, GPT 5.4) degrade documents by ~25% over long relay workflows across 52 domains. Degradation decelerates but never plateaus, and errors compound silently, remaining undetected in spot-checked outputs.
DELEGATE-52 shows weaker LLMs degrade documents through visible deletion, while frontier models degrade through subtle corruption that preserves surface integrity. This shift makes frontier failures harder to detect and potentially more dangerous at workflow scale.
DELEGATE-52 shows that agentic tool access fails to improve performance on long-horizon document tasks. The degradation mechanism originates upstream in the model's judgment about what to change, not in editing interface limitations.
Fixed seeds and zero temperature replicate the same output repeatedly, but that output remains one draw from the model's probability distribution. McDonald's omega testing across 100 repetitions reveals that consistency does not equal reliability.
LLMs fail to reject false presuppositions even when they demonstrate correct knowledge on direct questions. Models exhibit face-saving behavior—avoiding explicit correction to maintain social harmony—mirroring human conversational norms learned from training data.
Show all 7 sources
Research identifies four mechanical safeguards: ordering unarguable checks before contestable ones, measuring correctness against human labels, hiding test data from proposers, and using planted cases as alarms. None requires the LLM itself to verify compliance.
Research shows reliable LLM agents externalize three cognitive burdens—memory (state persistence), skills (procedural components), and protocols (structured interaction)—into a harness layer rather than relying on model scale alone. The harness unifies these externalities and eliminates the need for the model to solve the same problems repeatedly.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- RRSI: Regularized Recursive Self-Improvement of Agent Harnesses
- LLMs Corrupt Your Documents When You Delegate
- Can You Trust LLM Judgments? Reliability of LLM-as-a-Judge
- Bilevel Coordinated Reflection: A Game-Theoretic Approach to Multi-Agent LLM Systems
- Demystifying Agent Skills: Why They Work-Until They Don't
- The Illusion of Diminishing Returns: Measuring Long Horizon Execution in LLMs
- Emergent Collusion in Long-Horizon LLM Agent Interaction
- LLM-as-a-Judge Is Not an Oracle: Why Self-Improving Agents Need Deterministic Guardrails