AI can become highly capable through self-play alone — but human feedback is what separates 'effective' from 'acceptable.'

This explores a division of labor: self-play is good at manufacturing raw competence — it discovers strategies by grinding against itself — but it has no idea which of those strategies humans would actually accept. Human-anchored rewards supply that second thing, the convention. The corpus repeatedly shows these two functions coming apart, and it's most legible in driving: thirty minutes of human demonstration works best not as the main teacher but as a thin regularizer bolted onto self-play RL, nudging an otherwise competent policy toward the equilibrium humans actually drive in Can human data steer self-play RL toward human-compatible behavior?. The self-play half finds a policy that wins; the human half picks which winning policy is socially legible. Tellingly, the convention signal is tiny — 2500x less data than imitation learning — which is exactly what you'd expect if its job is to constrain, not to teach.

The deeper reason the two must be separated is that pure self-improvement is structurally circular. A model checking its own work hits the generation-verification gap, diversity collapse, and reward hacking — it can get more fluent at its own conventions without getting more correct Can models reliably improve themselves without external feedback?. Every method that actually improves smuggles in an external anchor: a past model version, a third-party judge, a user correction, a tool's verdict. Competence can bootstrap; convention cannot bootstrap itself, because 'what counts as good' is precisely the thing the system can't see from the inside.

So the interesting move in the corpus is manufacturing a convention signal *without* handing over a human reward every step. Ctx2Skill runs a three-role loop — a Challenger that escalates difficulty, a Judge that gives binary verdicts, and skills that co-evolve in natural language — so the Judge stands in for the missing human standard while self-play generates the competence Can language models learn skills without human supervision?. But the same note warns it collapses without a generalization safeguard: strip the anchor too far and self-play just games its own judge. That's the reward-hacking failure mode seen from another angle — and one clean fix is to keep the human standard as a *gate* rather than a *reward*, using rubrics to accept or reject whole rollout groups instead of converting them to dense scores the model can exploit Can rubrics and dense rewards work together without hacking?. Convention gates; competence optimizes inside the gate.

Here's the thing you might not have expected: at enough scale, the convention scaffolding starts to become unnecessary for some skills. A 104B model needed hand-designed rewards to learn structured reasoning and self-verification, but a 1T model discovered those same strategies on its own under pure zero RL Does scale alone teach models to reason without hand-crafted rewards?. In other words, some of what we thought was 'convention we have to inject' turns out to be latent competence the model finds by itself once it's big enough — which reframes a lot of reward-engineering as scaffolding whose value is tied to model size. And it connects to a subtler point about what rewards even carry: scalar rewards encode *evaluation* (how good was that?) but discard *direction* (how should it change?), so a human-anchored signal is doing two separable jobs at once Can scalar rewards capture all the information in agent feedback?. Related lines worth pulling: agents can treat the consequences of their own actions as supervision without any external reward at all Can agents learn from their own actions without external rewards?, and models can even internalize the evaluator, learning to score themselves in the unused space after their output Can models learn to evaluate their own work during training? — each of which shifts where the competence/convention boundary actually falls.