Could an AI learn to judge its own work just by watching humans correct and critique it?
Can judgment interventions eventually become training data that machines learn to replicate?
This explores whether the corrections, critiques and evaluations people make when they step in to judge an AI's work can be turned into training material, so the machine eventually makes those calls itself.
This explores whether human judgment, meaning the critiques, verdicts and corrections people supply when they step in, can become something a model learns to reproduce. The short answer from the corpus is yes, partly. Some kinds of judgment transfer much better than others, though, and the way judgment is packaged matters more than how much of it you collect.
The clearest evidence that judgment can be absorbed comes from work on evaluation itself. When LLM judges are given reference answers and explicit instructions for using them, they become reliable enough to supervise a model's self-improvement about as well as a purpose-trained reward model Can reference examples make LLM judges reliable enough for self-improvement?. Another approach goes further and trains the model to grade its own output in the unused space after it finishes answering. The evaluator is folded into the model, so no outside judge is needed at runtime Can models learn to evaluate their own work during training?. Judgment doesn't even have to come from humans: a diverse group of peer models scoring each other can stand in for an external judge, while a single model rewarding itself tends to drift and collapse Can peer models replace external judges for reward signals?.
The surprise is in what form the judgment takes. A verdict alone, whether a score, a label or a thumbs-up, carries less than you'd expect. Models stuck on a performance plateau with numerical rewards improved once they were given written critiques explaining *why* an answer failed Can natural language feedback overcome numerical reward plateaus?. In argument-quality assessment, fine-tuning on labeled examples taught surface patterns that didn't carry over to new kinds of argument. Generalization improved only when the theoretical framework behind the judgments was spelled out Can models learn argument quality from labeled examples alone?. The pattern holds even without any training. Agents that write down reflections on their failures Can agents learn from failure without updating their weights?, and models that turn their own mistakes into explicit principles Does learning from mistakes improve in-context learning?, both improve because the *reasoning* behind a judgment is kept, not just the outcome.
There are two ceilings worth knowing about. First, agents trained on curated expert demonstrations can't get past what the curators thought to include Can agents learn beyond what their training data shows?. Replicated judgment inherits the blind spots of whoever supplied it. Second, judgments that barely distinguish one answer from another give a weak training signal, and models then collapse into generic, one-size-fits-all responses Why do language models collapse into generic templates?. Judgment is only useful as training data where it actually discriminates between answers. There is also evidence that much of this training *draws out* abilities the base model already has rather than installing new ones Do base models already contain hidden reasoning ability?. Judgment data may be better at steering a model toward what it can already do than at teaching it to see what it couldn't.
The takeaway: machines can learn to replicate judgment, but they learn the *reasons* far better than the *rulings*. If you want your interventions to become something a model can reproduce, explain why you made the call, not just what the call was. Even then, the model will mostly be limited by the range of situations its judges thought to cover.
Sources 10 notes
Anchoring LLM-judges to reference answers with explicit usage instructions improved judge accuracy by 6.8% and enabled self-improvement training via DPO to match finetuned reward model performance on AlpacaEval and Arena-Hard benchmarks.
Post-Completion Learning exploits unused sequence space after model output to train self-assessment capabilities during training while maintaining zero inference cost. The model learns to compute its own reward functions, internalizing evaluation rather than relying on external reward models.
Co-RL trains decoupled models using peer predictions as rewards, avoiding the bias and collapse of self-generated feedback. Heterogeneous cohorts consistently improve reasoning across benchmarks and often match ground-truth supervised training.
Critique-GRPO shows that models stuck on performance plateaus can generate correct solutions when given chain-of-thought critiques, revealing that numerical rewards lack critical information about why failures occur and how to improve.
Fine-tuning on labeled examples fails to transfer quality criteria to new argument types. Models learn surface patterns rather than principled criteria. Explicit instruction using frameworks like RATIO or QOAM significantly improves performance and generalization.
Show all 10 sources
Reflexion demonstrates that unambiguous environmental feedback (success/failure) enables agents to write useful self-diagnoses and improve across episodes without parameter updates. The binary signal prevents rationalization, and keeping reflections uncompressed preserves their usability.
LEAP demonstrates that models achieve better performance on reasoning and math tasks by intentionally erring on few-shot examples, reflecting on mistakes, and deriving explicit task-specific principles—without additional labeled data or fine-tuning.
Agents trained on static expert datasets cannot learn from their own failures or generalize beyond demonstrated scenarios because they never interact with environments during training. Competence is capped by what curators imagined, not by agent capacity.
When within-prompt reward variance is low, task gradients weaken and regularization dominates, pushing policies toward generic outputs. SNR-Aware Filtering—selecting high-variance prompts before updates—recovers performance across tasks and scales.
Five independent mechanisms—RL steering, critique fine-tuning, decoding changes, SAE feature steering, and RLVR—all elicit reasoning already present in base model activations. Post-training selects rather than creates reasoning; the bottleneck is elicitation, not capability acquisition.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Temporal Self-Rewarding Language Models: Decoupling Chosen-Rejected via Past-Future
- Critique-GRPO: Advancing LLM Reasoning with Natural Language and Numerical Feedback
- Self-Rewarding Language Models
- Meta-Rewarding Language Models: Self-Improving Alignment with LLM-as-a-Meta-Judge
- Post-Completion Learning for Language Models
- Echo Chamber: RL Post-training Amplifies Behaviors Learned in Pretraining
- Post-Training Large Language Models via Reinforcement Learning from Self-Feedback
- Agent Learning via Early Experience