When an AI improves itself automatically, why does fixing broken code beat endless tweaking of its settings?
Why do bug fixes carry more weight than hyperparameter tuning in pipelines?
This explores why, in automated research pipelines that iteratively improve an AI system, fixing broken code often gains more than tuning settings like learning rate, and what that says about where the real slack in a system sits.
This explores why, in automated research pipelines that iteratively improve an AI system, fixing broken code often gains more than tuning settings like learning rate. The sharpest evidence in the corpus is AutoResearchClaw, which reached a 411% F1 improvement on the LoCoMo benchmark through bug fixes, architectural changes, and prompt engineering. Each of those three, on its own, beat all the hyperparameter tuning combined Can autonomous research pipelines discover AI architectures that AutoML cannot?.
The likely reason is what each approach can reach. Hyperparameter tuning searches over knobs someone already exposed, and it assumes the pipeline underneath is doing what its designer thinks. A bug breaks that assumption, so no setting compensates for it. The note frames this as a capability gap: an autoresearch agent can read the code and reason about how the parts interact, while classic AutoML only turns dials on a system it can't inspect Can autonomous research pipelines discover AI architectures that AutoML cannot?. The question is less "which is bigger" than "which search space contains the mistake."
The same pattern shows up wherever people improve the system around a model instead of the model itself. Wrapping fixed weights in a better execution harness lifted several models on Terminal-Bench, including GPT-5.6 to 95.3%, with no retraining Can execution harnesses lift model performance without retuning weights?. A stronger model built inference-time harnesses that nearly doubled a weaker model's Theory-of-Mind scores. It did this mostly by moving unstable reasoning into deterministic code and routing tasks, which is essentially removing a flaky step Can a stronger model lift a weaker one at test time without retraining?. The Darwin Gödel Machine's gains on SWE-bench came from discovering better code editing and context management, which are code-level changes, not parameter tweaks Can AI systems improve themselves through trial and error?. When these harness improvements are evolved on data separate from the benchmark, they still carry over to unseen tasks. That suggests they fix real mechanisms and don't just overfit Can harness modules improve separately from benchmark data?.
There is a limit on the claim. The bug-fix-beats-tuning comparison comes from a single system, and the other notes show the same direction without repeating the head-to-head test. The advantage also depends on the domain. Autoresearch works when results give an immediate score, the code is modular, iteration is fast, and changes are version-controlled. Without those, the bottleneck is the environment and not the model's ability to spot bugs What makes a research domain suitable for autonomous optimization?. Bug fixes win in pipelines where a broken piece can be found, changed in isolation, and scored right away.
Sources 6 notes
AUTORESEARCHCLAW achieved 411% F1 improvement on LoCoMo through bug fixes, architectural changes, and prompt engineering—each individually exceeding all hyperparameter tuning combined. This demonstrates a categorical capability gap: autoresearch can read code and reason about system-level interactions; AutoML cannot.
StateM improves Terminal-Bench 2.1 accuracy across multiple models by optimizing execution systems around frozen weights. The same runbook transfers to newer models without modification, achieving 95.3% on GPT-5.6 and lifting DeepSeek-V4 Flash by 5.4 points.
A stronger model built inference-time harnesses that nearly doubled weaker model performance on Theory-of-Mind benchmarks without retraining, primarily by moving unstable reasoning into deterministic code and task-specific routing rather than encouraging extended reasoning.
DGM replaces formal proofs with empirical benchmarking and maintains an evolutionary archive of agent variants, achieving 2.5× improvement on SWE-bench and 2.2× on Polyglot by discovering capabilities like better code editing and context management.
ModularRSI evolves harness modules independently using contrastive trajectories on benchmark-disjoint data, showing consistent gains across unseen tasks and domains. The approach isolates mechanism-level improvements from task-specific adaptation by aggregating evidence across tasks before updating components.
Show all 6 sources
Autonomous research pipelines require immediate scalar metrics, modular architecture, fast iteration cycles, and version control. Domains lacking any property resist autoresearch regardless of LLM capability, because the bottleneck is environmental structure, not model power.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- DarwinX: Evolving Agent Harnesses Through Natural Selection
- Harness Updating Is Not Harness Benefit: Disentangling Evolution Capabilities in Self-Evolving LLM Agents
- Rethinking the Evaluation of Harness Evolution for Agents
- Scaling Laws for Agent Harnesses via Effective Feedback Compute
- HarnessDev: Can LLMs Create and Evolve Their Own Agent Harness?
- Bilevel Autoresearch: Meta-Autoresearching Itself
- OMNI-SIMPLEMEM: Autoresearch-Guided Discovery of Lifelong Multimodal Agent Memory
- RRSI: Regularized Recursive Self-Improvement of Agent Harnesses