The scoring system that judges an AI can itself become the thing holding the AI back — unless it's designed to grow.

This explores how controlled utility evolution — letting the evaluator itself change as the agent improves — keeps the scorer from becoming the thing that caps or corrupts progress. The core problem it solves: any fixed evaluator is a stationary target, and a stationary target invites two failure modes. It saturates (the agent maxes it out and further gains stop registering), and it gets reward-hacked (the agent learns to satisfy the letter of the metric rather than the intent). Both mean the evaluator, not the agent's actual capability, becomes the ceiling Why do fixed benchmarks fail as agents get stronger?.

The trick that prevents this is *controlled* evolution rather than free-for-all drift. The Red Queen Gödel Machine splits the search into epochs: within an epoch the criteria are held stable so the agent has a fair, gameable-in-good-faith target with per-epoch guarantees, but across epoch boundaries the utility itself moves forward, staying just ahead of the optimizer. This is what makes co-evolving the evaluator viable on tasks that have no clean automated grader at all — creative writing, proof generation — where a static verifier would either not exist or immediately be hacked Can evaluators improve alongside the agents they score?. Notably, co-evolved systems don't just avoid collapse; they match fixed-evaluator performance while spending fewer tokens, which suggests the agent and evaluator are sharing learning rather than fighting each other.

The deeper reason an evaluator becomes a bottleneck is the generation-verification gap: pure self-improvement is circular because a model checking its own work can't exceed its own blind spots, and the loop stalls out on diversity collapse and reward hacking Can models reliably improve themselves without external feedback?. Reliable self-improvement always smuggles in some external anchor. What controlled utility evolution does is turn the evaluator into a *moving* anchor instead of a fixed one — the non-stationarity is the external pressure that keeps the verification side from ossifying into the thing the generator learns to exploit.

There's a cross-cutting move here worth seeing: making the objective itself a first-class, evolvable object. A bi-level agent can compile natural-language goals into executable scoring functions and feed optimization results back into goal design, so objective formulation becomes part of the discovery loop rather than a fixed input handed down at the start Can agents evolve their own objectives during search?. And the evaluator can be given more room before it caps out — reward models that reason with chain-of-thought before scoring raise their own capability ceiling and scale with test-time compute, so the judge gets stronger as the problems get harder rather than flattening Can reward models benefit from reasoning before scoring?.

The thing you didn't know you wanted to know: 'controlled' is doing the load-bearing work, and it's a governance problem as much as an algorithmic one. If the utility can drift arbitrarily, you've just moved the reward-hacking one level up — now the agent games the evolution of the objective. That's why safe self-evolution treats prompts, tools, and evaluators as versioned, registered resources with auditable lineage and rollback: the point is to decouple *what* evolves from *how* it evolves, so a moving evaluator stays measurable and reversible instead of becoming an unaccountable moving target How can agent self-evolution be made safe and auditable?.