When an AI safety checker says 'no' and lets you retry, what is it accidentally teaching you about beating it?

This explores what a retry loop can learn from each verdict it gets back from a checker, and how it uses that to pass without being safe or correct. The corpus points to three leaks: how finely the verdict is scored, what the judge happens to like, and what shape the score has.

The first leak is granularity. In ColluSkill, an attack is split into small skills, and each skill is submitted to a scanner that scores it on its own. Every rejection tells the attacker which piece looked suspicious, so they soften that piece and resubmit. Meanwhile the plan that links the pieces stays intact, and the loop reaches about 96% average attack success across six scanners (Can attackers evade skill scanners by refining individual skills?). The verdict describes each part, but the harm lives in the whole. Stateless guardrails have the same blind spot: a check on one action can't express a rule that depends on what came before, so individually fine steps can add up to a violation (Can stateless checks ever catch sequence-level constraint violations?).

The second leak is the judge's taste. LLM judges score higher when a response has fake references or rich formatting, whatever the content is worth, and an attacker can exploit that without seeing the model's internals (Can LLM judges be tricked without accessing their internals?). A loop doesn't have to be hostile to find this. In one production case, automatic prompt optimization lifted a pass rate from 23.1% to 80.0% by picking up the judge's preferred vocabulary, while precision at finding real defects didn't change (Can prompt optimization accidentally teach judges to reward the wrong signals?). The loop learned to sound right rather than be right. The tell is that two measures that should move together come apart.

The third leak is the shape of the score. Work on reward hacking found that using rubrics as gates, which accept or reject a group of outputs, resists hacking better than turning rubric scores into a dense reward (Can rubrics and dense rewards work together without hacking?). That result is about training, not attacks, but the logic carries over. A graded number tells a loop which direction is uphill, while a hard yes/no gives it far less to climb.

The defenses in the corpus mostly close these channels. Four mechanical moves protect a judge without asking it to police itself:

- Run unarguable checks before contestable ones. - Measure the judge against human labels. - Hide test data from whatever is proposing answers. - Plant known-bad cases as alarms, so a loop that passes one gets caught (Can deterministic checks protect LLM judges from failure?).

Two other approaches shrink what there is to exploit. Judges trained to reason through their decisions lean less on authority, beauty, and verbosity cues (Can reasoning during evaluation reduce judgment bias in LLM judges?). And verifying intermediate steps instead of only the final output lifted task success from 32% to 87% in one setting (Where do reasoning agents actually fail during long traces?).