An AI's edit to a running system can look perfectly sensible and still fall flat. Why is sounding right not enough?

This explores why an edit that looks reasonable to a language model so often fails to help, or breaks things, once it is applied to something that actually runs. The corpus's sharpest answer is that plausibility and effect are different targets. In Does training editors on real outcomes beat prompting larger models?, prompted frontier models patched an agent's harness (the scaffolding that runs its tools and steps) by optimizing for edits that sound right, and their gains were unstable or lower. A 9B model trained with reinforcement learning on whether patches worked, and which reran each patch to check its impact, raised a frozen agent by 9.3 points across three tasks. A running system has behavior you can't read off the edit itself. Only running it tells you whether the change did anything.

Better tooling doesn't fix this, because the mistake happens before the edit is made. Can better tools fix LLM document editing errors? finds that giving models agentic tools doesn't make long-horizon document editing more reliable. The degradation starts in the model's judgment about what to change, not in a clumsy editing interface. The same pattern shows up when the thing being edited is the model itself. Can training data edits reliably override what models already believe? shows that training on synthetic documents adds new information predictably, but contradicting what the model already believes is unpredictable. Does synthetic document finetuning fail at larger scales? finds it couldn't reliably override reward hacking at the scales tested. A system that is already running has entrenched behavior, and a new edit has to collide with it rather than land on a blank slate.

The fixes in the corpus all make an edit earn its place through observed results. Does constraining edits make skill learning more stable? limits how much an agent can rewrite its own instructions per step, checks each change on held-out examples before accepting it, and keeps rejected edits as negative feedback. That beat letting the agent rewrite freely. Can AI systems improve themselves through trial and error? gives up on proving an edit is good and benchmarks it instead, keeping an archive of agent variants. That lifted SWE-bench performance 2.5×. Can executable programs replace transformer attention heads? shows the same discipline on a model's internals. Synthesized programs matched 99% of attention head behavior, but only the best-fit 30–40% of heads could be swapped out while question answering still held up. Once the edit target is executable, whether the edit works becomes a testable question.

Once the target is a live system, the edit and its checks also have to sit where the system actually looks. Can prompts alone reshape multi-agent workflows without system access? shows a crafted prompt biasing how a multi-agent workflow forms, before any defenses inspect it, and raising malicious success by up to 55 percent. A plausible input upstream shapes what runs downstream. Can governance rules embedded in runtime memory actually protect autonomous agents? reports safeguards worked better when written into the memory layer the agent consulted mid-task than when kept in external policy. Can a finite lifecycle model detect reward hacking across benchmarks? checks a run against its intended lifecycle rather than matching known bad patterns. Preventing bad edits is also harder to credit than it looks. Do authorization rules or restricted tools prevent test modifications? reports zero protected-test modifications, but because it bundled authorization rules with restricted tools, nobody can say whether edits were impossible or merely unchosen.

The corpus is stronger on what rescues edits (outcome feedback, bounded changes, validation gates) than on cataloguing the mechanics of how edits break running systems. What it shows is that a plausible edit fails because it was judged by how it reads instead of by what it does when run.