Before an AI update goes live, what should the rulebook it must pass actually say, and who checks it?

This explores what the outside-the-model standard should specify when it decides whether an update ships. Macaron-V1 treats that standard as the 'external contract' and argues that improvement lives in the loop of model, harness and contract, not in any single set of weights. In that view, weight updates are gated by audit and evaluation against the contract, which makes the loop the unit of improvement and release Where does model adaptation actually happen?. The corpus has no checklist for what goes in it. What it does have is evidence about what breaks when the contract is missing or weak, and that evidence points to a fairly concrete list.

First, the contract has to name who or what supplies the signal the model can't supply for itself. Pure self-improvement stalls because of the generation-verification gap (checking an answer is not easier than producing it), diversity collapse and reward hacking. The methods that reliably work all bring in an outside anchor: past model versions, third-party judges, user corrections or tool feedback Can models reliably improve themselves without external feedback?. The gap can be measured. It grows with model size but vanishes on factual tasks, so the contract can say domain by domain where self-checking is trusted and where an external check is required What limits how much models can improve themselves?. It shouldn't be strict everywhere. Self-play with majority-vote verification Can language models improve themselves without any external training data? and rewards built from the model's own confidence Can model confidence alone replace external answer verification? both work in reasoning domains without external references. That suggests the contract should demand outside anchors where verification is no easier than generation, and relax elsewhere.

Second, the contract should say what is being measured, and on whose hands. Harness quality can't be read off downstream task scores, because models differ sharply in building and evolving harnesses, and a harness's performance shifts a lot with the model that executes it Can language models build and maintain their own agent harnesses?. The ability to write useful harness edits is flat across model tiers, but the ability to benefit from them peaks in mid-tier models. Weak models fail to invoke the harness, and strong ones struggle to follow it faithfully Do stronger models always evolve harnesses better?. So the contract needs to pin down the executor. It also needs a baseline: harness scaling alone lifted frozen models on Terminal-Bench 2.1, and the same runbook transferred to newer models unchanged Can execution harnesses lift model performance without retuning weights?. My inference is that a claimed improvement should have to beat what the harness gives you with the weights held fixed.

Third, the audit has to be able to see quiet failures. Weaker models degrade documents by visibly deleting content, while frontier models corrupt them in ways that keep the surface intact, and that is harder to detect at workflow scale Does model capability change how documents degrade?. A contract that checks only that outputs look complete would pass exactly the failures that matter most as models get stronger. Turning an LLM into an agent already treats safety evaluation as its own stage, separate from training, because the surrounding system decides whether actions are grounded or hallucinated Can you turn an LLM into an agent by just fine-tuning?.

Finally, the contract should say what is versioned together. Prime Agent keeps state in four levels outside the weights, which lets it separate harness failures from model failures Can external state caches let models solve harder problems?. Metis folds memory into the backbone to avoid the decoupling failures that arise when external memory and the model optimize independently Should agent memory live inside the model backbone?. Either way, the contract has to state which parts can change in a release and which are held fixed. Put together, a contract states who judges, on which tasks, against which baseline, with which executor, with checks for silent corruption, and across which versioned pieces. Only one note in this corpus uses the term 'contract' directly, so treat this as a synthesis from the surrounding evidence, not a settled spec.