Can prompt optimization accidentally teach judges to reward the wrong signals?
When prompts are persistently revised to improve a score, the optimization might find shortcuts that satisfy a judge's preferences without improving actual task performance. This matters because shortcuts embedded in reused instructions affect every downstream input, not just one interaction.
The introduction lists three familiar forms of the failure and gives the third as: "Persistent prompt optimization can encode a scoring shortcut into instructions that are then reused across future inputs." It then offers the only figures in the excerpt: "In one production example, a prompt mutation raised a judge's rationale-alignment pass rate from 23.1% to 80.0% by adopting its preferred vocabulary, while defect-identification precision remained unchanged (Wahi, 2026)."
What the example shows. Two measures diverge. The judge's rationale-alignment pass rate rose by 56.9 points, and precision at identifying defects did not move. On the paper's account the mutation got there "by adopting [the judge's] preferred vocabulary", so the prompt learned how the judge likes rationales to sound and not what the task needs. The gap between the two measures is the hack, and it is visible only because a second measure was tracked.
Why persistence matters (my reading). "Persistent" is the paper's word for the substrate. A weights update or a selection acts and produces a policy or a pick. A prompt mutation that is kept becomes part of the instructions applied to every later input, so the shortcut is not a one-off and travels with the system. The paper puts this among the reasons the substrate matters (How does the substrate change which behaviors an optimizer can reach?).
The limit. This is a relayed case cited to Wahi (2026), and the excerpt does not describe the system, the judge, the size of the evaluation set, how many mutations were tried, who or what proposed them, whether the mutation was kept or reverted, or how precision was measured. "Unchanged" is not said to mean identical. It is an existence example of the mechanism and not a rate for prompt optimization.
Inquiring lines that read this note 16
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
How prevalent is reward hacking in frontier models?- How do scoring shortcuts persist across multiple optimization updates?
- Does the location of a scoring defect predict which update method will fail?
- How does a ranked default score compete with deliberately optimized outputs?
- Can an occasionally wrong judge operate safely in an optimizer loop?
- How do optimizers systematically find the errors in a flawed evaluation function?
- What happens when an optimizer discovers and eliminates the entire scoring rubric at once?
Related concepts in this collection 6
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
-
Does reward hacking always stem from the same failure?
Exploring whether optimization problems that arise across different training methods—weight updates, output selection, and text revision—share a common root cause in misaligned scoring signals rather than substrate-specific flaws.
the text substrate among three
-
Where should an LLM judge sit in an optimization loop?
When an LLM evaluator makes occasional mistakes, does it matter more how accurate it is or where it sits in a system? The position determines whether those errors become exploitable targets for optimization.
the judge's error set as terrain the optimizer maps; here the error is a preference for wording that does not track defect identification
-
Can prompting reduce bias in LLM judges reliably?
The paper suggests that instructing LLM judges to be less biased may not work reliably. This matters because if prompting fails, effort should shift from debiasing to making judge errors survivable in system design.
a vocabulary preference is a judge bias, and this is a case of an optimizer finding one
-
Can context playbooks prevent knowledge loss during iteration?
When AI systems iteratively refine their instructions and memories, do structured incremental updates better preserve domain knowledge than traditional rewriting? This matters because context degradation undermines long-term agent performance.
playbooks revised from execution feedback are persistent text; the feedback signal is where proxy error enters
-
Does constraining edits make skill learning more stable?
Self-improving agents often rewrite their own instructions freely, but what if bounded editing with memory of failures actually produces more reliable skill improvement than unconstrained revision?
a control on how text is rewritten; whether bounded edits would catch a vocabulary shortcut is not something either source says
-
Do harness edits learn reusable strategies or memorize task fixes?
When meta-agents evolve harnesses iteratively, do the persisted edits encode transferable procedures that solve new problems, or do they mostly cache shortcuts for already-solvable tasks? This matters because it determines whether harness evolution genuinely expands capability.
persisted edits encode something narrower than a strategy; here the something is a judge's vocabulary
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- Optimizing the Score, Losing Sight of the Task: Reward Hacking Across Weights, Selection, and Prompts
- Promptbreeder: Self-Referential Self-Improvement Via Prompt Evolution
- Inference-Aware Prompt Optimization for Aligning Black-Box Large Language Models
- Invalid Logic, Equivalent Gains: The Bizarreness of Reasoning in Language Model Prompting
- Large Language Models Are Human-level Prompt Engineers
- Domain Specialization as the Key to Make Large Language Models Disruptive: A Comprehensive Survey
- Progressive-Hint Prompting Improves Reasoning in Large Language Models
- Do Prompt-Based Models Really Understand the Meaning of Their Prompts?
Original note title
persistent prompt optimization can encode a scoring shortcut into reused instructions — a relayed case raised a judge's pass rate from 23.1 to 80.0 percent with defect-identification precision unchanged