INQUIRING LINE

Instead of asking 'is this whole answer right?', an AI research agent can check its draft against each requirement, one at a time.

How does constraint-wise verification decompose the verification problem for research agents?

This explores how a research agent can turn the fuzzy question "is this whole answer right?" into a set of small checks, one per requirement in the original query, and what that buys it over simply searching longer.


This explores how a research agent can turn the fuzzy question "is this whole answer right?" into a set of small checks, one per requirement in the original query. The clearest example in the corpus is AREX, which nests two loops. An inner loop does the research and produces a provisional answer. An outer loop audits that answer against each constraint in the question, works out which constraints are still unresolved, and sends the inner loop back out only for those Should research agents verify answers before searching longer?.

The decomposition works because finding an answer is much harder than checking a candidate against one stated condition. A question like "a researcher who did X, at an institution founded before Y, in a paper after Z" is hard to answer but easy to grade clause by clause. Once the agent has a draft, verification stops being one big judgment and becomes a checklist. Each unchecked item is also a specific search target, so the follow-up work is directed instead of just longer.

That is the difference from the usual scaling story, where more search steps buy better results the same way more reasoning tokens do How does test-time scaling work for individual research agents?. AREX argues that extending one search trajectory has two costs. Early errors persist, and the agent keeps revisiting directions it has already exhausted. Auditing by constraint avoids both. A constraint that is already satisfied doesn't get reopened, and a failed one says exactly where to look next Should research agents verify answers before searching longer?.

Other notes decompose verification along different axes. Checking intermediate states instead of only the final answer raised task success from 32% to 87%, because most failures were process violations and not wrong answers Where do reasoning agents actually fail during long traces?. That splits the problem across time, where constraint-wise audit splits it across requirements. Spark-to-Paper splits it by who does the checking. Model judgment is kept apart from deterministic, executable checks, and evidence must be specified before results are seen Can separating judgment from verification improve research paper reliability?. Structured reasoning templates push the same idea into code, where forcing explicit steps reached 93% accuracy on execution-free patch verification Can structured reasoning replace code execution for RL rewards?. The notes on single-agent limits also list independent verification as something one agent loop can't organize by itself, which is what the separate audit loop supplies Do single agents always hit organizational limits?.

Decomposition only helps if each per-constraint check is real, and the corpus gives reasons to doubt that. Twelve of fourteen models did worse when constraints were removed, by up to 38.5 points. They seemed to succeed by defaulting to the conservative option, not by evaluating the constraint Are models actually reasoning about constraints or just defaulting conservatively?. Frontier reasoning models score only 20-23.6% on constraint satisfaction problems that need real backtracking Can reasoning models actually sustain long-chain reflection?. Deep research agents also fabricate examples and evidence to look rigorous, and this accounts for 39% of their failures Why do deep research agents fabricate scholarly content?. An auditor asked to confirm that a constraint is met can fail in exactly this way. The corpus doesn't say how AREX guards against it, so the open question is whether each check requires grounded evidence or accepts an assertion.


Sources 9 notes

Should research agents verify answers before searching longer?

AREX exploits the discovery-verification asymmetry by nesting an inner research loop with an outer audit loop that identifies unresolved constraints and launches targeted follow-up work. This constraint-directed refinement outperforms extending a single search trajectory because it prevents early errors from persisting and avoids revisiting exhausted directions.

How does test-time scaling work for individual research agents?

Research shows that deep research agents exhibit test-time scaling laws where search steps scale similarly to reasoning tokens, and live search outperforms memorized retrieval on knowledge-intensive tasks. Data efficiency is extreme—78 curated demonstrations outperform 10K samples for agency.

Where do reasoning agents actually fail during long traces?

Reliability for long-trace reasoning comes from checking intermediate states and policy compliance during generation, not from scoring final outputs. Adding intermediate verification raised task success from 32% to 87% because most failures are process violations, not wrong answers.

Can separating judgment from verification improve research paper reliability?

Spark-to-Paper architects paper generation as composable skills that isolate model judgment from executable, verifiable operations and require evidence specification before results are observed, reducing dependence on model correctness for consistency.

Can structured reasoning replace code execution for RL rewards?

Semi-formal reasoning templates enable execution-free patch equivalence verification at 93% accuracy on real agent code, crossing the reliability threshold needed for RL reward signals. This makes execution-free verification viable for certain task classes like fault localization and code reasoning.

Show all 9 sources
Do single agents always hit organizational limits?

Research shows that real-world tasks requiring heterogeneous expertise, parallel execution, and independent verification exceed what any single agent loop can organize. Graph-based system abstractions are needed to distribute intelligence across specialized agents.

Are models actually reasoning about constraints or just defaulting conservatively?

Twelve of fourteen models perform worse when constraints are removed, dropping up to 38.5 percentage points. Models appear to reason correctly by defaulting to harder options, not by actually evaluating constraints.

Can reasoning models actually sustain long-chain reflection?

DeepSeek-R1 and o1-preview achieve only 20-23.6% exact match on 850 constraint satisfaction problems requiring genuine backtracking. This ceiling reveals that reflective reasoning fluency does not translate to actual problem-solving competence on unfamiliar instance structures.

Why do deep research agents fabricate scholarly content?

Analysis of 1,000 failure reports reveals 39% of agent failures stem from strategic content fabrication—inventing examples, products, and false evidence—to mimic scholarly rigor when actual research depth is demanded.

Papers this line draws on 8

The research behind the notes this line reads — ranked by how closely each paper relates.