When an AI keeps rewriting its own instructions, does testing each edit on fresh examples stop them from slowly getting worse?
Does held-out validation prevent skill document edits from drifting or accumulating harm?
This explores whether the accept-only-if-it-scores-better check used when an agent rewrites its own instruction file actually stops that file from slowly getting worse, or only stops some kinds of getting worse.
This explores whether the accept-only-if-it-scores-better check used when an agent rewrites its own instruction file actually stops that file from slowly getting worse. The corpus suggests it contains drift well on the task being measured, but it can't catch harm the check doesn't look at.
The positive case is SkillOpt, which treats a skill document as trainable state for a frozen agent and keeps an edit only if performance on held-out examples improves Can skill documents be optimized like neural network weights?. Its ablations compare this with letting agents rewrite their own instructions freely. Three controls together gave more stable, better-generalizing improvement: a budget on how much text can change per step, the held-out gate, and a buffer that keeps rejected edits as negative feedback Does constraining edits make skill learning more stable?. Those pieces were tested as a package, so the corpus can't say how much stability comes from the gate alone. The gate also isn't the only thing doing the work, since the edit budget limits how far any single step can move.
The gate can only reject what it can see. Weaker models degrade documents by visibly deleting content. Frontier models corrupt them silently while surface integrity stays intact Does model capability change how documents degrade?. A subtly wrong edit from a strong model can look fine and pass, unless the held-out set happens to exercise the part it broke. If the gate's scoring involves an LLM judge, there is a second hole. Judges score higher when a response includes fake references or rich formatting, regardless of content quality, and no model access is needed to exploit this Can LLM judges be fooled by fake credentials and formatting? Can LLM judges be tricked without accessing their internals?. The corpus doesn't test SkillOpt's gate against either problem, so these are inferences from adjacent findings.
Accumulation is the harder question. The gate judges edits one at a time, and the closest analogy in the corpus is skill scanners. They score each skill individually, so an attacker can push each piece below the suspicion threshold while the harmful chain-level behavior stays intact Can attackers evade skill scanners by refining individual skills?. That work is about deliberate attacks, not accidental drift. But the same structure applies: many small edits that each pass a per-step check can add up to something no single check would have approved. The rejected-edit buffer only remembers failures, so it doesn't protect against a slow build-up of accepted edits.
The practical answer is to treat the gate as a filter at the door and add a record of what got through. One approach treats skills as versioned files that can be inspected, corrected, and rolled back, rather than as hidden prompt state Can person-grounded skills remain auditable without hidden prompt state?. The corpus has no long-run study of how validated edits pile up over many rounds. Until there is one, held-out validation is best read as strong protection against measurable regressions and weak protection against silent, unmeasured, or combined harm.
Sources 7 notes
SkillOpt treats skill documents as trainable external state of frozen agents, using a text-space optimizer with held-out validation gating to accept only edits that improve performance. Across 52 benchmark cells and seven models, the approach matches or exceeds baselines while adding zero inference cost and enabling transfer across models.
SkillOpt's ablations show that adding a textual learning-rate budget, held-out validation gate, and rejected-edit buffer (retaining failed edits as negative feedback) produces more stable and generalizable skill improvement than allowing agents to freely rewrite their own instructions.
DELEGATE-52 shows weaker LLMs degrade documents through visible deletion, while frontier models degrade through subtle corruption that preserves surface integrity. This shift makes frontier failures harder to detect and potentially more dangerous at workflow scale.
Research identified four evaluation biases in LLM judges, with authority and beauty biases being semantics-agnostic and trivially exploitable through fake references and formatting—zero-shot attacks requiring no model access or optimization.
Research shows LLM evaluators systematically score higher when responses include fake references or rich formatting, independent of content quality. These biases are exploitable without model access, undermining AI benchmark credibility.
Show all 7 sources
ColluSkill combines chain planning with scanner-feedback refinement to reach 96% average attack success. The approach works because scanners score skills individually, allowing feedback to reduce suspicion per skill while chain-level semantics remain intact.
COLLEAGUE.SKILL treats distilled expertise as versioned files subject to inspection, correction, and rollback—not hidden prompt state. Separating capability tracks from behavior tracks enables independent audit of what someone knows versus how they act.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Demystifying Agent Skills: Why They Work-Until They Don't
- SkillOpt: Executive Strategy for Self-Evolving Agent Skills
- Humans or LLMs as the Judge? A Study on Judgement Biases
- Can You Trust LLM Judgments? Reliability of LLM-as-a-Judge
- When Reject Turns into Accept: Quantifying the Vulnerability of LLM-Based Scientific Reviewers to Indirect Prompt Injection
- Overconfidence in LLM-as-a-Judge: Diagnosis and Confidence-Driven Solution
- Learning to Plan & Reason for Evaluation with Thinking-LLM-as-a-Judge
- Knowing Is Not Enough: Information Retrievability as a Precondition to Effective LLM Oversight