Why do autonomous research systems release code but not verification artifacts?

The survey codes 26 full-text entries from a screen of 125 candidates (35 included works: 24 runnable systems plus two study or position works) on seven audit dimensions. Its main finding is a split between two kinds of release. "Code release is now common (83% of the 24 runnable systems), but the artifacts and checks that let a reviewer verify a result are not." Only 38% release "the seeds or execution traces needed to reproduce a run," and only 38% report any novelty-verification method. The 22-system LLM-era subset shows the same pattern. These are the authors' own codings of public records, and they call the rates "directional audit evidence."

The authors locate the problem in what evaluation measures. "Most reported evaluation reduces to task success or a reviewer score," while trustworthy research would also need evidence on validity, novelty, reproducibility, and selection, which are "rarely measured." Novelty is the sharpest case: 38% of systems report a novelty-checking step, yet "we found no system that reports independent validation that its novelty check is itself reliable." The contrast is drawn against older lineages. The Robot Scientist "logged hypotheses and provenance in machine-readable form, so each claim was auditable by construction," whereas LLM research agents inherited the generation machinery and not the checks: "the capability transferred; the verification did not."

This sits closest to Can AI verify research outputs as fast as it generates them?, which draws the generation-verification gap from a roadmap's qualitative findings. This excerpt counts one stretch of that gap and places it in release practice: publishing code does not make a claim checkable. The authors' answer is disclosure. Their reviewer checklist asks for seeds, traces, selection policy, and novelty-check method, which fits the governance reading in Does more automation actually hide rather than eliminate errors?. They also name independent, self-preference-robust agent review as an open problem, which is where Can inference scaling help reviewers catch errors humans miss? would need to be tested; the excerpt does not assess that work.

What the excerpt does not establish is how exact these rates are. The coding covers only computational AI/ML research, where "code, experiments, benchmarks, and write-ups can be inspected," so the sample says nothing about other fields. Second-coder agreement was 90% on artifact release but only 50–65% on autonomy level, novelty method, and selection disclosure. The reliability check used abstracts only, so it does not validate the full-text reading behind the headline figures, and the excerpt gives no selection-disclosure rate at all. The implication at the strength the evidence allows: in this corpus, code availability and verifiability diverge sharply, and the 83% versus 38% contrast is a fair directional signal. The size of the gap is not settled until the full-text second pass the authors name as the next step.