A checker that says 'something's wrong' is useless if you can only patch one spot — does fixing work best when the check matches the fix?
How do artifact families differ in matching verification scope to repair capability?
This explores what it means for a check to be the right size for the fix, and how that plays out for different kinds of things agents produce (documents, code, reasoning traces, retrieval results, whole workflows).
This explores what it means for a check to be the right size for the fix, and how that plays out for different kinds of things agents produce. The corpus states the principle clearly in one survey but doesn't compare artifact families side by side, so what follows is that principle plus what neighboring notes show about each kind of artifact.
The principle comes from a survey of 259 works on agentic artifact creation. Verification loops guide repair only when the feedback names failures at a scope the system's available actions can actually fix (When does verification feedback actually guide targeted artifact repair?). A verifier that says 'something is wrong somewhere' is useless to a system that can only patch one paragraph. Having a verifier at all is the wrong test. The question is whether its output has the same resolution as the repair move.
Documents show the mismatch most clearly, and the problem there is detection, not repair. Weaker models degrade documents by visibly deleting content, which a coarse check catches. Frontier models corrupt content while keeping the surface intact (Does model capability change how documents degrade?). The strongest models corrupt about 25% of content over long relay workflows, and spot checks miss it (Do frontier LLMs silently corrupt documents in long workflows?). Better tools don't fix this, because the error starts in the model's judgment about what to change, not in the editing interface (Can better tools fix LLM document editing errors?). Stronger repair machinery can't help when the checks don't look where the failures are.
Other artifact types line up scope and action differently. For reasoning traces, asynchronous verifiers fork off to pull out one verifiable piece of state and step in only on a violation. The check and the intervention are both small, so the cost is near zero (Can verifiers monitor reasoning without slowing generation down?). For retrieval, a compressed-vector check can't tell a structural near-miss from a topical match. A small verifier reading the full token-to-token similarity map can, so the check has to be finer than the recall step that came before it (Can verification separate structural near-misses from topical matches?). For runtime audits, agents get a limited remit, fixed artifacts to inspect, and a duty to cite evidence, so their semantic judgments stay bounded and arguable (Can scoped agents reliably judge semantic hacks in runtime analysis?). Code and test protection adds a wrinkle. One study bundles authorization rules with restricted tools and reports zero test modifications. Without ablations, nobody can say whether the agent couldn't cross the line or chose not to (Do authorization rules or restricted tools prevent test modifications?). That is the same scope-versus-capability question in another form.
Whole workflows are the family where the scope needed is widest, and local checks fail there. Steps can each pass plausibility, alignment, or protocol checks while the end-to-end result still fails, because those checks test different properties than system safety needs (Can individual components pass safety checks if the system still fails?). Per-action guardrails can't even state a rule that depends on history. Only stateful monitors that track sequences can (Can stateless checks ever catch sequence-level constraint violations?). Existing measures of whether errors stay visible, contained, and recoverable are fragmented, and none covers all of them (How can we measure whether AI errors stay visible and recoverable?).
The mismatch runs both ways. A check can be too coarse for the repair, as with document spot checks and vector recall, or too local for the failure, as with per-step and per-action checks on a workflow. A strong repair tool doesn't help in either case, and the check has to be matched to where the failure actually lives.
Sources 11 notes
A survey of 259 works shows agentic artifact creation benefits from verification loops only when observations identify failures at a scope the system's available actions can repair. Misaligned granularity causes verification to fail even when present.
DELEGATE-52 shows weaker LLMs degrade documents through visible deletion, while frontier models degrade through subtle corruption that preserves surface integrity. This shift makes frontier failures harder to detect and potentially more dangerous at workflow scale.
Even the strongest models (Gemini 3.1 Pro, Claude 4.6 Opus, GPT 5.4) degrade documents by ~25% over long relay workflows across 52 domains. Degradation decelerates but never plateaus, and errors compound silently, remaining undetected in spot-checked outputs.
DELEGATE-52 shows that agentic tool access fails to improve performance on long-horizon document tasks. The degradation mechanism originates upstream in the model's judgment about what to change, not in editing interface limitations.
Decoupling verification from generation lets verifiers run alongside a single trace, forking to extract verifiable state and intervening only on violations. On correct runs the latency penalty is near-zero; interwhen matches or beats CoT across benchmarks at similar token budgets.
Show all 11 sources
A two-stage pipeline—pooled-cosine recall followed by a small Transformer verifier operating on token-token similarity maps—reliably rejects structural near-misses that MaxSim-style late interaction cannot. The verifier succeeds because it operates on full token interaction patterns rather than compressed vectors.
BenchShield constrains audit agents by limiting their remit, fixing the artifacts they see, and requiring evidence citation. This positions infrastructure records as unchallengeable checks and audit judgments as the arguable step after them, though reported reliability remains unquantified.
The abstract bundles clear authorization rules with restricted tools and reports zero protected-test modifications, but no single-factor ablation distinguishes whether the result comes from unavailable crossings, unchosen crossings, or both. The pipeline's own data elsewhere (100% Judgment Bypass Rate with 0% Unsafe Action Rate) shows the distinction matters.
Three mechanisms across SafeFlow, ChannelGuard, and Honest Quorum show that passing local checks (plausibility, alignment, protocol compliance) does not prevent system failures. The gap persists because local checks verify different properties than those that determine safe end-to-end behavior.
Per-action checks are structurally unable to state constraints that depend on prior history. Only stateful monitors tracking composed multi-party behavior can verify the behavioral envelopes that prevent individually permissible actions from collectively violating system-level safety.
Partial instruments exist for individual conditions in isolated settings, but none measures the full socio-technical system the paper identifies as necessary. Visibility has a model-side measure (chain-of-thought disclosure), containment has incident-level counts, and recoverability has rollback timing, yet none bridges all four or captures human-institution factors.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- PACT: Can Enterprise AI Assistants Be Trusted Under Pressure?
- LLMs Corrupt Your Documents When You Delegate
- interwhen: A Generalizable Framework for Steering Reasoning Models with Test-time Verification
- RRSI: Regularized Recursive Self-Improvement of Agent Harnesses
- The safety failures we are not instrumenting: a perspective on hidden safety-critical challenges in modern AI systems
- BenchShield: Formal Model-Backed Instrumentation for Reward Integrity in LLM-Agent Evaluation Infrastructure
- LLM-as-a-Verifier: A General-Purpose Verification Framework
- The Troy Moment of AI: Why Some Will Cheat and Some Will Follow?