Tweaking the prompts and tools around an AI is cheap and easy to read, but how does that compare to retraining the model itself?
How does editing the harness layer differ from updating model weights?
This explores how changing the scaffolding around a model (its prompts, tools and the code that runs it, called the harness) differs from changing the model itself by training its parameters. The corpus has no head-to-head comparison, but the two sides fit together into one picture.
This explores how changing the scaffolding around a model (its prompts, tools and the code that runs it, called the harness) differs from changing the model itself by training its parameters. The corpus has no head-to-head comparison. Read together, though, harness edits look cheap, readable and shallow, while weight updates look deep, hard to inspect and prone to side effects.
Start with what a harness edit does. When systems evolve their own harnesses automatically, the edits are sensible and well-motivated across the prompt and tool layers. But most of them just persist fixes an agent could have rediscovered in a single attempt, so they cache what is already within reach and rarely turn hard failures into successes Do harness edits learn reusable strategies or memorize task fixes?. Reinforcement learning changes the model's own machinery. It touches only 5–30% of parameters, but in structured, nearly full-rank subnetworks that look almost the same across random seeds Does reinforcement learning update only a small fraction of parameters?. It works largely by suppressing wrong reasoning paths, which unlocks strategies the base model couldn't reach What actually changes inside a model during RL training?. On this evidence, weights can change what a model can do, while the harness mostly changes how reliably it uses what it already has.
A harness is also advice the model has to choose to follow. The ability to write useful harness edits is flat across model tiers, but the ability to benefit from them is an inverted U. Weak models fail to invoke the harness at all, and strong models struggle to follow its instructions faithfully Do stronger models always evolve harnesses better?. Harness gains are also easy to overcount, because some of the improvement is just extra computation. Only gains beyond a matched-budget test-time search baseline can be credited to the harness design How should we measure gains from automatic harness evolution?.
Weight updates cost more in collateral damage. RL training can collapse a pretrained model onto one dominant format and suppress the others Does RL training collapse format diversity in pretrained models?. Training on problems that are too hard teaches degenerate shortcuts that contaminate existing capabilities Do overly hard RLVR samples actually harm model capabilities?. Binary correctness rewards also push models toward confident guessing and hurt calibration Does binary reward training hurt model calibration?. A harness edit can't corrupt what the model knows, because it never touches it.
The choice isn't binary, and the middle options are worth a look. Proxy-tuning steers a model at decoding time and leaves its weights alone. It closes most of the alignment gap and preserves knowledge better than direct fine-tuning, which corrupts knowledge storage in lower layers Can decoding-time tuning preserve knowledge better than weight fine-tuning?. Representation finetuning edits hidden states in a frozen model and beats LoRA on parameter efficiency by 10–50x Can editing hidden representations beat weight updates for finetuning?. Prompts and tools, decoding, hidden representations and weights form a spectrum. The further inside the model you edit, the more you can change what it can do, and the more you risk breaking what it already does well.
Sources 10 notes
Analysis of evolved harness trajectories shows rational, well-motivated edits across prompt and tool layers, but most persist fixes an agent could rediscover in a single rollout. Gains remain limited because memorized shortcuts cache what's already within reach rather than converting hard failures into successes.
Across seven RL algorithms and ten LLM families, RL induces intrinsic parameter sparsity of 5–30% without explicit regularization. Critically, these sparse updates are nearly full-rank and nearly identical across random seeds, indicating structural rather than arbitrary parameter selection.
RL modifies only 5-30% of parameters through structured subnetworks, with negative reinforcement (blocking wrong solutions) appearing as the primary mechanism. Training follows a predictable two-phase pattern: procedural consolidation precedes strategic exploration, unlocking reasoning strategies inaccessible to base models.
Model capability to produce useful harness edits stays constant across tiers, but capacity to actually benefit from those edits follows an inverted U-shape, peaking in mid-tier models. Weak models fail to invoke harnesses; strong models struggle with faithful instruction-following.
Automatic harness evolution must be compared against task-level test-time search under equal feedback and inference budgets. Only gains beyond what matched search achieves are attributable to the harness design itself, not just more computation.
Show all 10 sources
Controlled experiments show RL consistently amplifies one format distribution from pretraining within the first epoch while collapsing alternatives. The winning format depends on model scale, not necessarily performance, and is largely hidden when starting from proprietary pretrained models.
Training on nearly-impossible problems causes models to learn degenerate shortcuts rather than genuine reasoning, and these shortcuts contaminate pre-existing capabilities. Group-relative normalization treats rare accidental successes as high-advantage trajectories, reinforcing answer repetition and computation-skipping instead of sound reasoning patterns.
Binary correctness rewards incentivize high-confidence guessing because they don't penalize confident wrong answers. Adding the Brier score as a second reward term mathematically guarantees joint optimization of accuracy and calibration without trade-off.
Proxy-tuning closes 88-91% of the alignment gap while surpassing direct fine-tuning on knowledge tasks by leaving base model weights untouched. Direct fine-tuning corrupts knowledge storage in lower layers, whereas proxy-tuning applies distributional shifts that primarily affect reasoning and style.
ReFT learns task-specific interventions on frozen model representations rather than updating weights, with LoReFT (low-rank linear subspace variant) dramatically outperforming LoRA across reasoning, instruction-following, and NLU benchmarks while using far fewer parameters.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Echo Chamber: RL Post-training Amplifies Behaviors Learned in Pretraining
- Harness Updating Is Not Harness Benefit: Disentangling Evolution Capabilities in Self-Evolving LLM Agents
- Rethinking the Evaluation of Harness Evolution for Agents
- HarnessDev: Can LLMs Create and Evolve Their Own Agent Harness?
- Ouroboros: A Self-Developing Frontier Coding Agent with Reviewed Core Evolution
- Mechanistically Interpreting the Role of Sample Difficulty in RLVR for LLMs
- Reinforcement Learning for Reasoning in Large Language Models with One Training Example
- DarwinX: Evolving Agent Harnesses Through Natural Selection