How does the substrate change which behaviors an optimizer can reach?
Different optimization substrates—weights, selection, text—differ in which behaviors they can produce, what information they preserve, and what can be inspected or constrained. Understanding these differences matters because defenses or risk estimates may not transfer between substrates.
The conclusion follows the shared-mechanism claim with its qualifier: "The substrate matters because it changes which behaviors are reachable, what information persists, and what can be inspected or constrained." The abstract gives a nearby list of factors: "how reachable behavior, optimization budgets, and persistent adaptation shape exposure to proxy error." The two lists overlap on reachable behavior and persistence. The abstract adds optimization budgets and the conclusion adds inspection and constraint, and the excerpt does not reconcile them.
How I read the three properties against the three substrates (the excerpt states the properties and not this mapping). Reachable behavior is what the optimizer can land on: for selection, the pool of candidates; for weights, what the parameters can be moved to; for text, what instructions can elicit. Persistence is what the optimization leaves behind: the paper names its third substrate "persistent text", and a mutated prompt is reused on later inputs (Can prompt optimization accidentally teach judges to reward the wrong signals?). Inspection is whether a person can see what was learned, and the relayed case has the shortcut sitting in a prompt's vocabulary. I do not read the sentence as ranking the substrates, and the discussion says outright that "one substrate is always safest" is not assumed.
Why it matters. A defense or a risk estimate carries assumptions about all three properties. Moving it from one substrate to another silently changes them. That is the framing under Which reward hacking defenses actually transfer across training substrates?.
The vault already holds one pair that reads this way, on my reading and not the paper's. Why does majority voting outperform more complex inference methods? credits agreement across samples with needing no verifier that can be wrong when it selects among them, and Does self-consistency reliably reward correct answers during training? finds the model learning to satisfy that same agreement once it becomes the training signal. Selecting among what a model already generated cannot move what it generates, and training on the signal can, which is the reachable-behavior property above. Neither note frames it that way, and "robust" in the first is accuracy against Best-of-N and not exposure to a scorer's blind spots.
The limit. The sentence is a conclusion with no argument in the excerpt, and nothing says the three properties are exhaustive. The vault's survey note argues that a prompt or memory update is "cheap and reversible" while a weight update is "expensive and risky"; those are cost and reversibility properties, and this sentence concerns what each substrate reaches, keeps and exposes to inspection. Whether reversibility helps against a shortcut that reads as an improvement is logged as a tension.
Inquiring lines that read this note 1
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
How prevalent is reward hacking in frontier models?Related concepts in this collection 6
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
-
Does reward hacking always stem from the same failure?
Exploring whether optimization problems that arise across different training methods—weight updates, output selection, and text revision—share a common root cause in misaligned scoring signals rather than substrate-specific flaws.
the claim this qualifies: one mechanism, three substrates that differ
-
Do self-improving agents really split into two distinct loops?
Explores whether modern self-improving agents can be understood through a clean abstraction separating fast scaffold updates from slow model weight updates, and whether this framework actually explains the field's recent progress.
its "cheap and reversible" versus "expensive and risky" contrast is about cost; this is about reach, persistence and inspection
-
How can agent self-evolution be made safe and auditable?
As agents begin updating their own prompts and tools, how can we track these changes, measure their effects, and safely reverse problematic updates? This matters because untracked evolution leads to unmaintainable systems and makes regressions impossible to diagnose.
inspection and constraint for the text substrate, made operational as lineage and rollback
-
Can distance alone rank which substrates resist reward hacking?
Does the amount a method changes a model reliably predict how exposed it is to evaluator errors? The paper tests whether a single distance metric can universally order vulnerability across weights, selection, and prompts.
the formal companion: what a distance measure leaves out
-
Why does majority voting outperform more complex inference methods?
Simple majority voting across independent samples often matches or beats sophisticated alternatives like Best-of-N and sequential revision. What makes this basic approach so hard to beat for reasoning models?
the selection half of the pair the body reads: agreement used to choose among fixed samples, credited with needing no verifier that can be wrong
-
Does self-consistency reliably reward correct answers during training?
Self-consistency initially correlates with correctness, but as models train on this signal, do they eventually learn to maximize consistency itself rather than accuracy? When does this proxy reward stop working?
the weights half: the same agreement as the training signal, which the model learns to satisfy; the pairing as one device on two substrates is the vault's reading
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- Optimizing the Score, Losing Sight of the Task: Reward Hacking Across Weights, Selection, and Prompts
- Choosing the Right Weights: Balancing Value, Strategy, and Noise in Recommender Systems
- DarwinX: Evolving Agent Harnesses Through Natural Selection
- Instruction Tuning for Large Language Models: A Survey
- It's Not What You Say, It's How You Say It: Evaluating LLM Responses to Expressions of Belief
- Performative Thinking? The Brittle Correlation Between CoT Length and Problem Complexity
- From Entropy to Epiplexity: Rethinking Information for Computationally Bounded Intelligence
- The Missing Layer of AGI: From Pattern Alchemy to Coordination Physics
Original note title
the substrate matters because it changes which behaviors are reachable, what information persists and what can be inspected or constrained — the paper's reason not to treat weights, selection and text alike