INQUIRING LINE

When an AI builds the scaffolding around itself, does it fit only the model that runs it, or work for any?

Do models co-adapt their harnesses to specific executor strengths?

This explores whether a harness (the tools, routing and code wrapped around a model) built by one model ends up tuned to the particular model that runs it, called the executor, or whether it stays a portable wrapper.


This explores whether a harness built by one model ends up tuned to the particular model that runs it, or stays a portable wrapper. The corpus suggests both are true for different layers of the harness. None of these notes tests co-adaptation head-on, though. The evidence comes from experiments that happen to swap executors.

The clearest sign of executor-fit is that the same harness performs very differently depending on who runs it. Can language models build and maintain their own agent harnesses? finds that harnesses an LLM builds swing sharply in performance when the executor changes. That is why the authors say harness quality can't be read off downstream task scores. Fit also isn't simply 'stronger is better'. In Do stronger models always evolve harnesses better?, models of every tier are about equally good at writing useful harness edits, but the ability to benefit from those edits peaks in mid-tier models. Weak executors often fail to invoke the harness at all. Strong ones struggle to follow its instructions faithfully. A harness has a sweet-spot executor: capable enough to use it and not so strong that it drifts from it.

When a stronger builder deliberately targets a weaker executor, its design choices look like compensation for specific gaps. In Can a stronger model lift a weaker one at test time without retraining?, the builder nearly doubled a weaker model's Theory-of-Mind scores. It did this mostly by moving unstable reasoning into deterministic code and adding task-specific routing, not by asking for longer reasoning. The same pattern shows up elsewhere. Why does autoregressive generation fail at constraint satisfaction? argues that symbolic solvers help because they supply the ability to retract a bad guess, which autoregressive generation lacks. When can weak models match strong model performance? shows that many weak-model attempts only match a strong model when an external check, such as a test or a proof, picks the right answer. The harness pieces that work tend to fill a hole the executor has, which is co-adaptation in spirit.

Other findings push the opposite way and say much of a harness is portable plumbing. In Can execution harnesses lift model performance without retuning weights?, the same runbook carried over to newer models unchanged. It reached 95.3% on GPT-5.6 and lifted DeepSeek-V4 Flash by 5.4 points. Can agent harnesses be automatically optimized across many environments? found four mechanisms (action execution, context compaction, observation handling, delegated reading) that cut token traffic by 44.7–49.0% with comparable performance, and calls harness gains orthogonal to model improvements. Can harness modules improve separately from benchmark data? evolves harness modules on data separate from the benchmarks and gets gains on unseen tasks and domains.

The gap in the collection is a direct comparison: a harness tuned to one executor's profile against a generic one, then run on other executors. Housekeeping layers like context handling and action execution seem to travel across models. The layers that compensate for a model's particular weaknesses, such as its unstable reasoning, are the ones most likely to be fitted to it.


Sources 8 notes

Can language models build and maintain their own agent harnesses?

Research shows that LLMs vary sharply in building harnesses across domains, struggle to retain useful intermediate updates during evolution, and produce harnesses whose performance shifts dramatically with different executors—demonstrating that harness quality cannot be inferred from downstream task scores alone.

Do stronger models always evolve harnesses better?

Model capability to produce useful harness edits stays constant across tiers, but capacity to actually benefit from those edits follows an inverted U-shape, peaking in mid-tier models. Weak models fail to invoke harnesses; strong models struggle with faithful instruction-following.

Can a stronger model lift a weaker one at test time without retraining?

A stronger model built inference-time harnesses that nearly doubled weaker model performance on Theory-of-Mind benchmarks without retraining, primarily by moving unstable reasoning into deterministic code and task-specific routing rather than encouraging extended reasoning.

Why does autoregressive generation fail at constraint satisfaction?

The performance ceiling on constraint satisfaction problems is not a model-quality issue but an architectural limitation: autoregressive transformers cannot retract emitted tokens, while CSP solvers fundamentally depend on discarding invalid partial assignments. Symbolic solver integration works because it supplies what the architecture lacks.

When can weak models match strong model performance?

Sampling alone amplifies coverage but cannot select correct solutions. Reliable performance matching requires external soundness signals—tests, proofs, or type checks—that convert latent correct proposals into actual selections.

Show all 8 sources
Can execution harnesses lift model performance without retuning weights?

StateM improves Terminal-Bench 2.1 accuracy across multiple models by optimizing execution systems around frozen weights. The same runbook transfers to newer models without modification, achieving 95.3% on GPT-5.6 and lifting DeepSeek-V4 Flash by 5.4 points.

Can agent harnesses be automatically optimized across many environments?

Cross-environment harness optimization yielded four mechanisms (action execution, context compaction, observation handling, delegated reading) that reduced token traffic by 44.7–49.0% while maintaining comparable performance on a 51-task benchmark, suggesting harness-level gains are orthogonal to model improvements.

Can harness modules improve separately from benchmark data?

ModularRSI evolves harness modules independently using contrastive trajectories on benchmark-disjoint data, showing consistent gains across unseen tasks and domains. The approach isolates mechanism-level improvements from task-specific adaptation by aggregating evidence across tasks before updating components.

Papers this line draws on 8

The research behind the notes this line reads — ranked by how closely each paper relates.