If AI agents keep a shared log of every experiment they run, is that enough to improve, or must something read it?
Does shared experimental state alone explain progress or is an analyzer needed?
This explores whether a shared record of what agents have tried (results, lineage, failures) is enough to make a group of research agents improve, or whether something has to read that record and decide what it means.
This explores whether a shared record of what agents have tried is enough to drive progress, or whether something has to interpret it. The corpus leans toward the record being necessary but not sufficient. No note tests the two head-to-head, though, so the evidence is indirect.
The best case for shared state alone is a run where thirteen language-model workers had no central planner. They wrote to an append-only Git graph of results and lineage. Over 12 days they made 1,703 contributions and closed 62% of the gap to a trained baseline, and later sessions could build on earlier ones without reconstructing them Can decentralized agents coordinate research without a central planner?. The record made progress cumulative. What it supplied was continuity, not judgment. Each worker still had to decide which branch was worth extending, and the note credits the lineage without separating out that judgment. So it's unclear how much of the 62% came from the record and how much from the workers reading it well.
The notes where something explicitly does the reading suggest the reader carries real weight. In AutoResearchClaw, every failed experiment goes through a pivot-or-refine decision that shapes the next attempt. Ablations show this mechanism drives completion, separately from reasoning or verification Can experiment failures drive progress instead of stopping it?. A log of failures sits there whether or not anyone decides what to do with it. AlphaEvolve makes a similar point with a lighter analyzer. Cheap, objective automated evaluators keep an evolutionary loop running long enough to produce real discoveries Can machine feedback sustain discovery at test time?. The analyzer can be as simple as a scoring function or as rich as a decision policy. Either way, the shared state is the substrate rather than the engine.
Sharing has a ceiling, and how you read the state changes what it says. Pooling observations never reduces a coalition's ability to tell decoys from real objects, but it only weakly increases it, so it can stay the same Does sharing observations help coalitions detect decoys better?. Sharing raises what can be known without guaranteeing that anyone knows it. Reading granularity matters too. Step-level confidence catches reasoning breakdowns that global averaging hides Does step-level confidence outperform global averaging for trace filtering?. The exploration-exploitation trade-off shows up when you measure at the token level and nearly vanishes in hidden-state analysis Is the exploration-exploitation trade-off actually fundamental?. The same state read by a different analyzer gives a different verdict.
The question also applies to us as observers, because saying state explains progress needs its own analyzer. Harness-evolution gains only count beyond what a matched-budget test-time search would have achieved How should we measure gains from automatic harness evolution?. Benchmark gains and genuine reasoning activation can come apart Can genuine reasoning activation coexist with contaminated benchmarks?. Recording whole trajectories moves the comparability problem into a higher-dimensional space rather than solving it Do interactive evaluations actually solve the benchmark comparison problem?. That shift from final answers to interaction sequences still needs a scoring procedure on top How should we evaluate agent behavior beyond final answers?. BenchShield makes the same move by having an operator issue a claim from recorded infrastructure evidence, not just reading off a score Can infrastructure evidence replace terminal scores in benchmark validation?. Records don't interpret themselves. Progress seems to need both memory and judgment, and the collection has not yet cleanly measured how much credit each deserves.
Sources 11 notes
Thirteen language-model workers with no central planner used a shared Git DAG to develop a weight-transfer method over 12 days, producing 1,703 contributions and closing 62% of the gap to a trained baseline. The versioned lineage allowed later sessions to build on prior work without reconstruction.
AutoResearchClaw's pivot-or-refine loop routes every failure through a decision process, making failure inform the next attempt rather than stop execution. Component ablation shows this mechanism drives completion and is distinct from reasoning or verification.
AlphaEvolve demonstrates that automated evaluators can sustain evolutionary loops long enough to produce real discoveries—faster algorithms, optimized hardware designs, and improved training methods. The key is that cheap, objective verification closes the generation-verification gap where discovery becomes computationally feasible.
Mathematical analysis shows that when agents share their observations, the coalition's capacity to distinguish decoys from genuine objects cannot decrease—it stays the same or improves. This means defenders cannot rely on isolation to hide decoys from coordinated observers.
Local step-level confidence catches reasoning breakdowns that global averaging masks and enables early stopping before traces complete. This approach achieves comparable accuracy gains to naive majority voting with far fewer generated traces, proving trace quality matters more than quantity.
Show all 11 sources
Hidden-state analysis using Effective Rank metrics shows near-zero correlation between exploration and exploitation, revealing the trade-off emerges only at token level. VERL demonstrates simultaneous enhancement achieving 21.4% accuracy gains on Gaokao 2024.
Automatic harness evolution must be compared against task-level test-time search under equal feedback and inference budgets. Only gains beyond what matched search achieves are attributable to the harness design itself, not just more computation.
RLVR activates genuine reasoning patterns through RL training while benchmark improvements may reflect data memorization on contaminated datasets. These operate at different measurement levels and can coexist without contradiction.
Interactive evaluation relocates core problems—comparability, reproducibility, evidence-to-judgment mapping—into higher-dimensional space rather than solving them. The field needs design protocols and shared standards, not format adoption, to make trajectory scoring interpretable.
Evaluation of agentic systems shifts evidence from final responses to full interaction sequences, and scoring procedure from correctness alone to process quality, recoverability, coordination, and robustness. This pattern appears across multiple agent benchmarks as a coherent design move.
BenchShield enables benchmark operators to issue claims about valid task completion grounded in recorded infrastructure evidence rather than terminal scores alone. This shifts from a single number to a verifiable claim about whether an agent followed the intended evaluation path.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Bilevel Autoresearch: Meta-Autoresearching Itself
- BenchShield: Formal Model-Backed Instrumentation for Reward Integrity in LLM-Agent Evaluation Infrastructure
- AutoResearchClaw: Self-Reinforcing Autonomous Research with Human-AI Collaboration
- Interactive Evaluation Requires a Design Science
- Local Coherence or Global Validity? Investigating RLVR Traces in Math Domains
- Dream-RSI: Recursive Self-Improvement through Evolving Worlds
- Beyond Final Scores: A Systematic Evaluation of Agents for Long-Horizon AI Research and Development
- OMNI-SIMPLEMEM: Autoresearch-Guided Discovery of Lifelong Multimodal Agent Memory