Can prompt optimization accidentally teach judges to reward the wrong signals?
When prompts are persistently revised to improve a score, the optimization might find shortcuts that satisfy a judge's preferences without improving actual task performance. This matters because shortcuts embedded in reused instructions affect every downstream input, not just one interaction.
The introduction lists three familiar forms of the failure and gives the third as: "Persistent prompt optimization can encode a scoring shortcut into instructions that are then reused across future inputs." It then offers the only figures in the excerpt: "In one production example, a prompt mutation raised a judge's rationale-alignment pass rate from 23.1% to 80.0% by adopting its preferred vocabulary, while defect-identification precision remained unchanged (Wahi, 2026)."
What the example shows. Two measures diverge. The judge's rationale-alignment pass rate rose by 56.9 points, and precision at identifying defects did not move. On the paper's account the mutation got there "by adopting [the judge's] preferred vocabulary", so the prompt learned how the judge likes rationales to sound and not what the task needs. The gap between the two measures is the hack, and it is visible only because a second measure was tracked.
Why persistence matters (my reading). "Persistent" is the paper's word for the substrate. A weights update or a selection acts and produces a policy or a pick. A prompt mutation that is kept becomes part of the instructions applied to every later input, so the shortcut is not a one-off and travels with the system. The paper puts this among the reasons the substrate matters (How does the substrate change which behaviors an optimizer can reach?).
The limit. This is a relayed case cited to Wahi (2026), and the excerpt does not describe the system, the judge, the size of the evaluation set, how many mutations were tried, who or what proposed them, whether the mutation was kept or reverted, or how precision was measured. "Unchanged" is not said to mean identical. It is an existence example of the mechanism and not a rate for prompt optimization.
Inquiring lines that read this note 33
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
How should evaluation expand to capture deployment realities?- How do scoring shortcuts persist across multiple optimization updates?
- How can a second performance metric reveal shortcuts that a single metric would hide?
- Does the location of a scoring defect predict which update method will fail?
- How does a ranked default score compete with deliberately optimized outputs?
- Can an occasionally wrong judge operate safely in an optimizer loop?
- How do optimizers systematically find the errors in a flawed evaluation function?
- What signals could refinement loops exploit in defense verdict systems?
- What happens when monitors themselves become targets for optimization?
- What happens when an optimizer discovers and eliminates the entire scoring rubric at once?
- Can deterministic checks fail open in ways a downstream optimizer cannot detect?
- How should tutor safety violations be ordered from gross to subtle?
- Can a prompt mutation exploit a judge's vocabulary preferences without improving actual performance?
- How much does instruction prompt design control what alignment target an AI annotator enforces?
- Why does grouping feedback by shared correction pattern improve prompt generalization?
- Why did prompt engineering emerge as a job title?
- How much do small wording changes in prompts affect what brands AI tools recommend?
- Can gradient-free prompt optimization beat manual prompt engineering at scale?
- Can persistent prompt optimization encode a scoring shortcut into reused instructions?
- What makes prompt sub-components so interconnected they require constant rework?
- What makes a prompt update cheaper and more reversible than a weight update?
- How do slow weight updates and fast prompt updates interact in self-improvement?
- How does supervised finetuning change what prompting techniques actually accomplish?
Related concepts in this collection 6
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
-
Does reward hacking always stem from the same failure?
Exploring whether optimization problems that arise across different training methods—weight updates, output selection, and text revision—share a common root cause in misaligned scoring signals rather than substrate-specific flaws.
the text substrate among three
-
Where should an LLM judge sit in an optimization loop?
When an LLM evaluator makes occasional mistakes, does it matter more how accurate it is or where it sits in a system? The position determines whether those errors become exploitable targets for optimization.
the judge's error set as terrain the optimizer maps; here the error is a preference for wording that does not track defect identification
-
Can prompting reduce bias in LLM judges reliably?
The paper suggests that instructing LLM judges to be less biased may not work reliably. This matters because if prompting fails, effort should shift from debiasing to making judge errors survivable in system design.
a vocabulary preference is a judge bias, and this is a case of an optimizer finding one
-
Can context playbooks prevent knowledge loss during iteration?
When AI systems iteratively refine their instructions and memories, do structured incremental updates better preserve domain knowledge than traditional rewriting? This matters because context degradation undermines long-term agent performance.
playbooks revised from execution feedback are persistent text; the feedback signal is where proxy error enters
-
Does constraining edits make skill learning more stable?
Self-improving agents often rewrite their own instructions freely, but what if bounded editing with memory of failures actually produces more reliable skill improvement than unconstrained revision?
a control on how text is rewritten; whether bounded edits would catch a vocabulary shortcut is not something either source says
-
Do harness edits learn reusable strategies or memorize task fixes?
When meta-agents evolve harnesses iteratively, do the persisted edits encode transferable procedures that solve new problems, or do they mostly cache shortcuts for already-solvable tasks? This matters because it determines whether harness evolution genuinely expands capability.
persisted edits encode something narrower than a strategy; here the something is a judge's vocabulary
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- Optimizing the Score, Losing Sight of the Task: Reward Hacking Across Weights, Selection, and Prompts
- Promptbreeder: Self-Referential Self-Improvement Via Prompt Evolution
- Inference-Aware Prompt Optimization for Aligning Black-Box Large Language Models
- Invalid Logic, Equivalent Gains: The Bizarreness of Reasoning in Language Model Prompting
- Large Language Models Are Human-level Prompt Engineers
- Domain Specialization as the Key to Make Large Language Models Disruptive: A Comprehensive Survey
- Self-Taught Optimizer (STOP): Recursively Self-Improving Code Generation
- Large Language Models Cannot Self-Correct Reasoning Yet
Original note title
persistent prompt optimization can encode a scoring shortcut into reused instructions — a relayed case raised a judge's pass rate from 23.1 to 80.0 percent with defect-identification precision unchanged