Why does a self-improving AI get further when its rules are written as code instead of plain English?

This explores why changing where or how a self-improving system's rules are written down (in plain English, in code, or as checks built into the loop) changes how much the system can improve itself. The clearest direct evidence comes from STOP, where a language model repeatedly rewrites the program that wraps around it while its own weights stay frozen Can language models improve their own scaffolding without weight updates?. The researchers found that the form of the checking signal matters a lot. A constraint written as source code gave the loop more to work with than the same constraint written in plain English. One way to read this: a rule written in code can be run and checked, while a rule written in prose has to be interpreted, and the model can interpret it loosely.

Other notes in the collection point the same way. When agents are allowed to freely rewrite their own instructions, their skills drift and become unstable. SkillOpt gets better results by building the limits into the loop itself: a cap on how much can change per step, a held-out test that each edit must pass, and a buffer that keeps rejected edits as examples of what not to do Does constraining edits make skill learning more stable?. The ACE framework makes a related move. It updates its context in small structured increments instead of rewriting it wholesale, which stops useful detail from being lost over repeated passes Can context playbooks prevent knowledge loss during iteration?. The Darwin Gödel Machine goes furthest. It drops the requirement that a self-modification be formally proven to help, and instead tests each variant against benchmarks while keeping an archive of past versions. That change is what makes open-ended improvement workable at all Can AI systems improve themselves through trial and error?.

The less obvious lesson comes from work on guardrails. A rule only works if it sits somewhere that can actually see what it is about. A check that looks at one action at a time cannot even express a rule about sequences of actions, such as 'these steps are each fine, but together they cause harm.' Only a monitor that tracks history can Can stateless checks ever catch sequence-level constraint violations?. Self-improvement loops face the same problem. A rule placed in advisory text that the model reads before acting is a different kind of thing from a rule placed in a gate that judges the outcome afterward. So moving a constraint can change more than how strictly it is enforced. It can change whether the constraint is enforceable at all.

There is also a reason constraints work better outside the model's own text generation. Language models write one token at a time and cannot take back what they have already written, while constraint solving depends on throwing out partial answers that turn out to be invalid Why does autoregressive generation fail at constraint satisfaction?. Even frontier reasoning models score only about 20-23% on problems that require real backtracking Can reasoning models actually sustain long-chain reflection?. If a constraint lives only in the prompt, the model has to enforce it during generation, which is its weakest mode. Moving the constraint into code, a validator, or an archive gives the loop the ability to discard bad attempts, which the model lacks.

One caveat: STOP is the only source in the collection that directly compares the same constraint written in different forms. The other notes support the idea by analogy rather than by running that exact experiment.