SYNTHESIS NOTE
Can models learn to evaluate their own work during training?
Explores whether language models can internalize reward function computation as part of training, transforming external feedback into internal self-assessment capability without slowing inference.
Current training paradigms terminate learning at the end-of-sequence token, wasting the entire sequence space after model output completion. Post-Completion Learning (PCL) systematically exploits this neglected space. A temporary termination marker (<-- post-completion -->) creates a "post-thinking" space where models continue generating self-assessments and reward predictions during training, while inference stops at the marker — zero additional cost at deployment.
The core innovation is white-box reinforcement learning: the model explicitly learns to understand and compute reward functions, internalizing the reward model as its own evaluation capability. This transforms the model from "passive reward acceptance" (external reward signal tells it what's good) to "active self-evaluation" (it learns to compute quality assessments itself).
Implementation uses dual-track SFT: one track optimizes reasoning, the other optimizes evaluation capability. These are mixed with RL training for multi-objective hybrid optimization. The model learns both to solve problems and to assess its own solutions — but critically, only the problem-solving capability is active during inference. The self-evaluation is internalized during training, shaping the model's generation without requiring explicit self-assessment at inference time.
This addresses three limitations simultaneously:
- SFT's passive learning — models learn to mimic demonstrations without developing self-assessment ability
- RL's external dependency — reward models are opaque external components; PCL internalizes the evaluation
- Self-correction's inference cost — methods like Self-Refine require additional generation passes; PCL's self-evaluation is absorbed into training
The parallel with human cognition is direct: "Humans, after completing a task, often engage in self-reflection and quality assessment — this post-thinking process is crucial for improving future performance." PCL operationalizes this for LLMs.
This connects to What limits how much models can improve themselves? — PCL attempts to close the gap by training the verifier and generator as the same model, with the verification capability internalized rather than external. It also complements Does reflection in reasoning models actually correct errors? — PCL's self-evaluation is trained against ground-truth reward functions, not against the model's own prior outputs, potentially avoiding the confirmatory pattern.
Inquiring lines that read this note 156
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
How effectively can models learn from their own generated feedback?- What happens when models train on AI-generated content recursively?
- Why does online RL succeed where supervised training fails for self-correction?
- Why do error avalanches accelerate in self-training loops without verification?
- Does training on critiques of noisy responses produce deeper understanding than imitating correct ones?
- Can models learn better from critiquing errors than imitating correct responses?
- Why does self-generated training data outperform externally curated domain examples?
- Can the serving loop itself become the primary training data source?
- Can models learn to generate their own training examples effectively?
- How do instruction backtranslation and MAGPIE demonstrate self-generation principles?
- Can self-training drift be prevented by applying student compatibility filtering?
- Can trained models encode programs more complex than their data-generating process?
- What happens when models train on feedback from their own generations?
- How does error distribution during training affect a model's ability to self-correct?
- What cognitive capabilities do agents need to internalize social feedback?
- How do evaluative versus directive signals differ in next-state training?
- What training objectives could reduce completion bias in autonomous agents?
- Can an agent's internal probabilities serve as value signals across domains?
- Does self-play feedback improve skills created from the agent's own experience?
- Why does delegation training help models that work alone?
- Can self-improving agents become truly autonomous without intrinsic metacognition?
- Does learning community preferences as training rewards operationalize prediction without participation?
- Can reward-guided decoding replace weight fine-tuning for personalized alignment?
- Can preference learning fix the rigid output format problem better than supervised training?
- Can LLMs evaluate their own observations without external feedback?
- How do self-generated preference pairs from a strong teacher compare to human feedback?
- Can synthetic self-play data teach models when to disagree?
- How do implicit world models and self-reflection operationalize consequence-based learning?
- Why does self-correction during generation produce reliable labels without exemplars?
- How should training incorporate external critique versus encouraging self-correction?
- Why does external critique improve revision accuracy more than self-assessment?
- Why does external critique improve revision while internal self-assessment fails?
- What distinguishes intrinsic metacognition from extrinsic human-designed loops?
- Why does self-judgment of success or failure work without ground truth labels?
- Can external retrieval signals outperform internal self-assessment during revision?
- Does external critique guide revision better than internal self-assessment during model training?
- Why does self-critique fail without external verification signals?
- Why does externalizing bookkeeping raise effective feedback compute?
- Does the generation-verification gap define where self-rewarding actually works?
- How do reward models and self-improvement mechanisms interact in training?
- How do self-play and human-anchored rewards separate competence from convention?
- Can environment feedback alone provide dense credit without a teacher?
- Can applicability conditions and veto rules make self-training stable across substrates?
- How does self-improvement capability vary across memory, retrieval, and update tasks?
- Can self-ratings of output quality predict forecast performance?
- Why can't AI models internalize audiences the way human experts do?
- Can systems recognize and abstain on judgments rather than hallucinating preferences?
- Can evaluation criteria be reliably encoded in labeled data without ground truth standards?
- Can subjective tasks be delegated without human feedback loops?
- Can AI evaluation match human judgment quality in structured domain tasks?
- Does meta-judging improve evaluator quality better than temporal decoupling alone?
- Can polished language output substitute for the judgment it should express?
- Why does AI-improved task performance fail to transfer to independent work?
- Does AI-assisted performance transfer to independent task completion?
- What training signals would models need to learn reciprocal common-ground construction?
- What other internal model decisions beyond attention could be optimized directly?
- How can language models extract more value from fewer demonstrations?
- Why does self-critiquing actually reduce plan quality in language models?
- Does self-revision actually improve reasoning in large language models?
- When does self-reflection actually help reasoning models improve?
- Does reflection training actually teach models to self-correct their mistakes?
- Does self-reflection help models notice their own constraint violations?
- Does internal self-revision actually degrade reasoning accuracy in models?
- How does confirmatory reflection differ from corrective self-evaluation in models?
- How does metacognitive self-correction enable models to revise failed strategies?
- Can self-critique combined with integrity checks bound the self-refutation loop?
- What role does natural language play in breaking reinforcement learning performance plateaus?
- Does RL training redirect self-doubt into productive gap analysis?
- Do models actually self-assess their confidence or just confirm answers?
- Can unsupervised confidence-based training scale to domains beyond human evaluation reach?
- Why do reward models learn surface-level shortcuts instead of genuine quality assessment?
- Why does natural language feedback break performance plateaus that numerical rewards alone cannot?
- Can reward model training be automated without changing feedback mechanisms?
- Could reward signals incentivize active intent discovery over passive response generation?
- How do semantic reward shaping approaches compare to full critique models?
- Why do generative reward models produce more interpretable evaluations than scalar scores?
- Can model confidence signals replace explicit external reward functions?
- Why do reward models fail when they ignore the prompt context?
- How can reward structures teach models when to speak and when to stay silent?
- How do reward models benefit from extended thinking during evaluation scoring?
- Can environmental rewards directly refine natural language descriptions of actions?
- Can binary judge feedback replace external reward signals for skill learning?
- Why does self-segmentation into chunks-of-thought matter for reward models?
- How does in-context feedback integration differ from learned reward signals?
- What makes step-wise rewards denser than final-answer correctness signals?
- Does pairwise self-judgment avoid reward model scaling problems?
- How do internal model mechanisms escape token-level reinforcement signals?
- How do pairwise self-judgment and internal belief-shift replace verification differently?
- What makes current learned reward models fail across different domains?
- How do graduated phase rewards emerge complex dialogue behavior from simple objectives?
- How does credit assignment work across many sequential decision steps in language models?
- Can structured natural language feedback outperform scalar rewards in RL?
- Can emotion-grounded rewards replace coarse bonus signals in hierarchical dialogue RL?
- Can distillation and reward optimization happen in a single training loop?
- Can co-evolved critics truly circumvent static evaluator limitations in self-improvement?
- Can AI systems improve themselves without external feedback?
- How does this scoped definition relate to the survey's open-ended recursive self-improvement?
- Does weak exogenous anchoring like compilation checks suffice for safe self-improvement?
- Do external perspectives fix the self-evaluation bias in language models?
- Can language models accurately evaluate the quality of their own ideas?
- Does bidirectional attention improve language models as universal encoders?
- Can textual gradients generalize natural language feedback across computation graphs?
- How do internal representations compare to human cognitive structures?
- Can language models generate plausible latent thoughts without human annotation?
- Does the prediction unit shape what language models actually learn?
- Can language models accurately evaluate the quality of their own reasoning?
- What emerges in large language models that makes explicit value modeling necessary?
- How does training distribution shape what language models understand best?
- What is the comprehension-generation asymmetry in language models?
- Do larger language models show stronger self-preference in evaluation tasks?
- Can language models learn research intuition directly from outcome labels?
- Can models be trained to recognize their own generated text reliably?
- Can self-supervised methods replace human annotations for process reward models?
- Does reverse-curriculum learning approximate process supervision using only outcome signals?
- Can self-supervised process models replace human annotations at scale?
- Does self-supervised process supervision work for domains with ambiguous correctness?
- How much data do generative process reward models actually need?
- Do self-supervised process reward models scale better than human annotation?
- Can language models function as implicit process reward models through retrospection?
- What are the actual limits of sibling comparison versus trained process reward models?
- Can models learn both what and how to study through reinforcement learning?
- How does reinforcement learning on outcomes reinforce template-matching rather than computation?
- How does self-referential processing transfer to other reasoning tasks?
- How do language models infer their own mental states like humans do?
- Do models spontaneously develop self-reflection from minimal training signals?
- Can models detect when their own trajectory is on-policy versus off-policy?
- Which internal states can a language model access and report about itself?
- Can AI learn intrinsic motivation to assess its own relevance?
- Can AI systems generate and refine their own objective functions?
- How do current AI models perform when asked to specify their own goals?
- Can AI systems learn their own objectives through autoresearch?
- How misaligned are verbal reports from internal model computation?
- What does leveraging internal representations during training actually mean operationally?
- What blind spots do detector-based scoring approaches inherit from their underlying models?
- Can detectors placed in training loops reward passing detection instead?
Related concepts in this collection 4
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Concept map
Can models learn to evaluate their own work duri…
What limits how much models can improve themselves…
Does reflection in reasoning models actually corre…
Can model confidence work as a reward signal for r…
Do reward models actually consider what the prompt…
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
your link
semantically near
linked from elsewhere
-
What limits how much models can improve themselves?
Explores whether self-improvement has fundamental boundaries set by how well models can verify versus generate solutions, and what this means across different task types.
PCL addresses this by co-training generation and verification in the same model
-
Does reflection in reasoning models actually correct errors?
When reasoning models reflect on their answers, do they genuinely fix mistakes, or merely confirm what they already decided? Understanding this matters for designing better training and inference strategies.
PCL's evaluation is trained against external reward functions, potentially avoiding confirmatory bias
-
Can model confidence work as a reward signal for reasoning?
Explores whether using a language model's own confidence scores as training rewards can simultaneously improve reasoning accuracy and restore calibration that standard RLHF damages.
related: both use the model's own assessment capability as a training signal
-
Do reward models actually consider what the prompt asks?
Exploring whether standard reward models evaluate responses based on prompt context or just response quality alone. This matters because if models ignore prompts, they'll fail to align with what users actually want.
PCL internalizes reward computation, potentially avoiding the prompt-insensitivity problem
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- Post-Completion Learning for Language Models
- Temporal Self-Rewarding Language Models: Decoupling Chosen-Rejected via Past-Future
- Post-Training Large Language Models via Reinforcement Learning from Self-Feedback
- Learning to Reason without External Rewards
- LLM Evaluators Recognize and Favor Their Own Generations
- PretrainZero: Reinforcement Active Pretraining
- Self-Rewarding Language Models
- Meta-Rewarding Language Models: Self-Improving Alignment with LLM-as-a-Meta-Judge
Original note title
post-completion learning uses the ignored post-eos space to internalize self-evaluation during training with zero inference cost