Can AI design the scaffolding around itself — tools, memory, rules — better than people can, and where does that actually work?

This explores whether AI-built or AI-evolved harnesses beat human-designed ones in particular domains. The collection doesn't have a domain-by-domain scoreboard of that head-to-head. What it does have is evidence that models vary a lot in how well they build harnesses, some early results where automated harness optimization pays off, and several warnings about why "beats humans" is harder to establish than it sounds.

The closest note says that LLMs "vary sharply" in building harnesses across domains, so the honest answer is "some domains, unevenly." It also finds that a harness's quality can't be read off the downstream task score, because performance shifts dramatically depending on which model executes it (Can language models build and maintain their own agent harnesses?). Models also struggle to keep useful intermediate updates while evolving a harness. This treats harness design as its own skill, separate from solving the task.

The strongest evidence of machine-made harness gains is about efficiency. Auto-research loops run across many environments discovered four mechanisms (action execution, context compaction, observation handling, delegated reading). Together they cut token traffic by roughly 45–49% with comparable performance on a 51-task benchmark (Can agent harnesses be automatically optimized across many environments?). Separately, optimizing the execution system around frozen weights lifted several models on Terminal-Bench 2.1, including a 5.4-point gain for DeepSeek-V4 Flash (Can execution harnesses lift model performance without retuning weights?). Terminal and agentic-task settings look like the current sweet spot. Neither note is framed as a comparison against a human-designed baseline, though, so they show that harness optimization works, not that it beats human designers.

Three notes explain why that comparison is hard. Gains only count as design value if they beat test-time search given the same feedback and compute (How should we measure gains from automatic harness evolution?). Self-evolving harnesses can memorize their training tasks, so in-distribution wins shrink out of distribution (Does harness self-improvement memorize tasks instead of learning broadly?). The nearest thing to a cross-domain result is evolving harness modules separately on benchmark-disjoint data, which produced consistent gains on unseen tasks and domains (Can harness modules improve separately from benchmark data?).

The question may also be framed the wrong way round. Model strength doesn't predict who designs well or who benefits. The ability to produce useful harness edits is flat across model tiers, while the ability to benefit from them peaks in mid-tier models, because weak models fail to invoke the harness and strong ones struggle to follow it faithfully (Do stronger models always evolve harnesses better?). Safety harnesses show the same pattern: one strict enough for one model over-blocks another, so domain and model jointly decide what works (Should safety harnesses be customized for each deployment?). The better question may be which model-and-deployment pairs a harness suits, since no single domain has been shown to favor machine designers.