Does benchmark score predict a coding agent's self-improvement capacity?

The Huxley-Gödel Machine (HGM) paper identifies what it calls the "Metaproductivity–Performance Mismatch": when a self-improving coding agent searches a tree of self-modifications, favoring the child with the highest coding-benchmark score is misleading, because "a high-scoring agent may produce unproductive descendants, while a lower-scoring one seeds lineages that achieve greater long-term gains." The paper reports this as an empirical observation ("we empirically observe that immediate benchmark performance is an unreliable predictor of CMP") distinct from the method it then proposes to fix it — the two should not be collapsed into one claim.

The fix is clade-level metaproductivity (CMP), a metric "inspired by Huxley's notion of clades as lineages of common ancestry" that aggregates the benchmark performance of an agent's descendants rather than scoring the agent itself. The paper's Theorem 1 argues something stronger than a useful heuristic: under its stated Assumption 1 (the self-improvement process is judged only by the final agent's evaluation score, with repeatable evaluation trials), access to a true CMP oracle "suffices to imitate the Gödel Machine" — the original Gödel Machine's formal-proof-based acceptance rule for self-modifications, which is theoretically optimal but practically unusable because the proofs rarely exist. HGM is the practical algorithm that estimates CMP from partial, clade-aggregated outcomes and uses Thompson sampling to decide which agent in the tree to expand next, decoupling expansion from evaluation for asynchronous search. On SWE-bench Verified and Polyglot it reports higher-quality agents than prior methods at lower CPU-hour cost, and the agent it evolves on SWE-bench Verified with GPT-5-mini transfers to human-level performance on SWE-bench Lite with GPT-5 — a generalization result, not the metaproductivity claim itself.

This directly contradicts the search heuristic in Can AI systems improve themselves through trial and error?, whose "key assumption" — per that note — is that "improvement on coding benchmarks indicates better coding capabilities, which in turn indicates better ability to self-modify." HGM's paper names DGM and SICA explicitly as systems that "assume that higher software benchmark scores correspond to greater self-improvement capacity," and argues the mismatch between immediate score and descendant productivity undermines exactly that assumption, even though both systems operate in the same scaffold-editing regime — modifying code and prompts around a frozen model, not retraining weights. The fix (archive/clade of variants as stepping stones) is structurally similar between DGM and HGM; what changes is the statistic used to decide which stepping stone to expand.

The excerpt does not show how large or frequent the mismatch is outside this coding-agent benchmark setting, nor does it establish that CMP estimation (as opposed to the true oracle) reliably avoids the same mismatch it diagnoses in benchmark scores — the paper's own framing treats CMP as an estimate guided by Thompson sampling, not a solved measurement problem. If the mismatch generalizes, it implies that any tree-search or evolutionary self-improvement loop that selects which agent to expand based on that agent's own immediate score — rather than its lineage's eventual productivity — is optimizing the wrong signal, a concern that would extend beyond coding agents to any self-modifying system scored by a proxy benchmark.