SYNTHESIS NOTE
Topics›Evaluations›this note

Where does model adaptation actually happen?

Does improvement come from updating model weights alone, or does it require versioning the entire loop—harness, contract, and audit trail together? This matters for how we design and release systems that need to keep improving.

Synthesis note · 2026-09-25 · sourced from Evaluations

Macaron-V1 states its position directly in the discussion: "Adaptation is not the property of a single model; it is the property of the loop the model sits in." The abstract makes the unit of improvement a pair. Adaptation is pursued through "recursive improvement of versioned model-harness pairs, where experience from one configuration is evaluated under an external contract and used to construct its successor." The thing that gets better across generations is a model plus the harness it runs in, not weights alone.

The mechanism is a recursive self-improvement cycle. According to the discussion, the cycle generates harder tasks, audits trajectories under a versioned harness configuration, evaluates candidate Harness Context Protocols (HCPs), "and only then" uses selected trajectories to update weights. Weight updates come last and are gated by an audit and an evaluation against a contract that sits outside the model. The introduction gives the motivation. Post-training effectiveness is "closely coupled" to the environment where a model is trained, evaluated, and deployed, and those conditions "continue to evolve over time" as knowledge, domains, and interaction state change. If the environment moves, the training target moves with it, so the paper treats the loop as the object to design.

The second half of the design applies the same reasoning to collaboration. The Mixture-of-LoRA architecture freezes a base model, composes specialist LoRA adapters (chat, agent, coding, and GenUI in the shipped Venti model), and selects one LoRA per user turn. The paper calls collaboration "an architectural affordance" and not a multi-agent framework layered on top. Because the base is shared and adapters are portable, a registry could admit specialists trained by different teams or personalized for different users.

Against the nearest notes, this is a third framing of the same shift away from the bare model. Can agents adapt without pausing service to users? splits adaptation by speed. The Macaron-V1 excerpt describes one gated cycle and does not say whether it has a fast path. Where does agent reliability actually come from? makes the harness the unit of reliability, and Macaron-V1 adds that the harness is versioned and improved alongside the weights. Should we evaluate deployed agents as whole environments instead? argues for the same widening at the level of evaluation, while Macaron-V1 widens the unit of training and release.

The excerpt is explicit about its limits. "The released checkpoint is one snapshot and does not demonstrate improvement across generations," and the routing interface is a proposal that "tests only the four shipped specialists." In its own words, "Neither bet is settled by Macaron-V1." The excerpt reports no benchmark numbers, no ablation of the loop, and nothing on the external contract's content beyond its being versioned. What it supports is a design claim: if adaptation lives in the loop, then what to inspect and version is the harness, the contract, and the audit trail, and the released harness is offered for that purpose. Whether the loop actually improves successive generations is not shown here.

Inquiring lines that read this note 5

This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.

How does harness optimization generalize across different model architectures and domains? How do capability benchmark scores systematically misrepresent true model abilities?

Related concepts in this collection 5

This note in its neighbourhood — explore the map, then jump to a related concept in the list below.

Concept map
15 direct connections · 97 in 2-hop network ·medium cluster Open in graph ↗

Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph

your link semantically near linked from elsewhere

Related papers in this collection 8

Papers most semantically related to this note, ranked by cosine similarity in the embedding space.

Original note title

adaptation is a property of the loop the model sits in, not of a single model — versioned model-harness pairs improved under an external contract