Teams keep tuning the code around their AI, then upgrade the model, and those hard-won fixes quietly stop mattering.

This explores why improvements built into the scaffolding around a model (prompts, tools, control flow) so often fail to carry over when the underlying model is swapped or upgraded. The corpus has no direct measurement of how often updates disappear. Several notes do converge on one explanation: most harness edits answer one model's particular failures, so they lose their reason to exist when the model changes.

The first cause is that many edits are memorized fixes, not general strategies. An analysis of evolved harnesses found the edits were well-motivated, but most just persist fixes an agent could have rediscovered in a single attempt. They cache what was already within reach Do harness edits learn reusable strategies or memorize task fixes?. A cache keyed to one model's blind spots breaks in one of two ways. A better model doesn't share those blind spots, so the edit becomes redundant. A different model has different blind spots, so the edit misses them. A related note says an update's effect depends on the context that produced it: the parent model's state, the data, the training stage and the evaluation criteria. Promoting a new model also rewrites the baseline that later evidence is measured against, so past success shouldn't count as permission to reuse an update. It suggests checking applicability and running bounded trials first Should past update success guide future model changes?.

The second cause is that the payoff from a harness depends on where the model sits, even when the edit itself is fine. Models of every tier are about equally good at producing useful harness edits. The ability to benefit from them follows an inverted U, peaking in mid-tier models. Weak models often fail to invoke the harness at all, and strong models struggle to follow its instructions faithfully Do stronger models always evolve harnesses better?. Safety harnesses show the same fit problem: one strict enough for one model over-blocks another Should safety harnesses be customized for each deployment?. One note goes further and says adaptation isn't a property of a model snapshot. It lives in the versioned loop that links model, harness and an external contract, so swapping the model changes the loop itself Where does model adaptation actually happen?.

Not every update disappears, and the survivors share a pattern. Edits that fix the plumbing tend to last, and edits that patch a model's reasoning quirks tend not to. Four mechanisms found by scaling harness search across many environments (action execution, context compaction, observation handling and delegated reading) cut token traffic by about 45–49% at comparable performance, and the note calls these gains orthogonal to model improvements Can agent harnesses be automatically optimized across many environments?. One execution runbook transferred to newer models without modification Can execution harnesses lift model performance without retuning weights?. Evolving modules separately, on data disjoint from the benchmark and aggregating evidence across tasks before each update, is another way to keep mechanism-level gains apart from task-specific ones Can harness modules improve separately from benchmark data?. Some systems also guard against loss directly: DarwinX evolves harness variants under a 'preserve-and-extend' contract, an explicit rule for keeping earlier gains while adding new ones Can frozen models improve by evolving their harnesses?.

The practical upshot is to treat a harness edit as a hypothesis tied to the model that produced it, and re-test it on the new model instead of assuming it transfers. Some edits deserve to disappear. Harness gains should be compared against plain test-time search run with the same feedback and compute budget, and only the margin beyond that is attributable to the harness design How should we measure gains from automatic harness evolution?. An edit that was only buying extra computation, or patching a weakness the new model no longer has, was never a durable improvement.