SYNTHESIS NOTE
Topics›Novel Architectures›this note

Can models learn to evaluate their own work during training?

Explores whether language models can internalize reward function computation as part of training, transforming external feedback into internal self-assessment capability without slowing inference.

Synthesis note · 2026-02-23 · sourced from Novel Architectures

Current training paradigms terminate learning at the end-of-sequence token, wasting the entire sequence space after model output completion. Post-Completion Learning (PCL) systematically exploits this neglected space. A temporary termination marker (<-- post-completion -->) creates a "post-thinking" space where models continue generating self-assessments and reward predictions during training, while inference stops at the marker — zero additional cost at deployment.

The core innovation is white-box reinforcement learning: the model explicitly learns to understand and compute reward functions, internalizing the reward model as its own evaluation capability. This transforms the model from "passive reward acceptance" (external reward signal tells it what's good) to "active self-evaluation" (it learns to compute quality assessments itself).

Implementation uses dual-track SFT: one track optimizes reasoning, the other optimizes evaluation capability. These are mixed with RL training for multi-objective hybrid optimization. The model learns both to solve problems and to assess its own solutions — but critically, only the problem-solving capability is active during inference. The self-evaluation is internalized during training, shaping the model's generation without requiring explicit self-assessment at inference time.

This addresses three limitations simultaneously:

  1. SFT's passive learning — models learn to mimic demonstrations without developing self-assessment ability
  2. RL's external dependency — reward models are opaque external components; PCL internalizes the evaluation
  3. Self-correction's inference cost — methods like Self-Refine require additional generation passes; PCL's self-evaluation is absorbed into training

The parallel with human cognition is direct: "Humans, after completing a task, often engage in self-reflection and quality assessment — this post-thinking process is crucial for improving future performance." PCL operationalizes this for LLMs.

This connects to What limits how much models can improve themselves? — PCL attempts to close the gap by training the verifier and generator as the same model, with the verification capability internalized rather than external. It also complements Does reflection in reasoning models actually correct errors? — PCL's self-evaluation is trained against ground-truth reward functions, not against the model's own prior outputs, potentially avoiding the confirmatory pattern.

Inquiring lines that read this note 159

This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.

Can synthetic data preserve necessary diversity to prevent model collapse? Should agent evaluation measure execution quality beyond task success? Does generative AI replace coordination with individual productivity? How can reward models fail to detect reward-seeking disguised as alignment? What prevents or enables error accumulation in self-training systems? Do AI explanations represent actual model reasoning or constructed post-hoc narratives? Does AI assistance enhance or undermine professional skill development? When do additional thinking tokens stop improving reasoning performance? What reasoning limitations and biases do language models systematically exhibit? Why do human-AI teams underperform compared to their individual components? Can language models develop genuine communicative grounding with humans? Why does self-revision increase model confidence despite degraded accuracy? Does reinforcement learning create new reasoning capabilities or refine existing ones? How can model evaluations remain robust against gaming and evaluator drift? Why doesn't LLM knowledge translate to improved reasoning outputs? Should reward models emphasize process understanding or only final outcomes? What training data structures and signals produce robust learning across distributions? Do benchmark score improvements actually reflect genuine AI capability gains? How do different reward signals affect model reasoning and calibration differently? How can we distinguish genuine model self-knowledge from behavioral performance? How does AI writing assistance affect reader perception of writer authenticity? Does evolutionary optimization at test time improve model performance effectively? Do language models genuinely understand or primarily match statistical patterns? Do language models learn underlying linguistic structure or merely surface patterns? What explains the gap between LLM semantic understanding and production? How do different post-training structures affect learning outcomes? How does scaling test-time compute compare to increasing model size? What conversation mechanics enable genuine human-AI communicative participation? When should retrieval be triggered to augment language model reasoning? What external mechanisms enable reliable self-improvement in language models? Do language models truly understand their encoded knowledge or merely retrieve patterns? What determines whether visible reasoning traces improve task performance? Can AI systems autonomously pursue unstated objectives or circumvent design constraints? Does training data format shape reasoning more than domain content? Can AI agents learn to improve performance from experience? How do mechanistic features and representations drive model behavior? What actually drives chain-of-thought performance gains in reasoning tasks? How can language models effectively manage long-context memory without quality degradation? How can agents learn and retain useful knowledge across diverse environments? Can harness optimization rival model scaling in effectiveness? Does chain-of-thought reasoning faithfully reflect actual model reasoning processes? Can we reliably detect when models engage in reward hacking? Does reward hacking in training cause emergent misalignment? How bounded is recursive self-improvement in current AI systems? What systematic biases do LLM judges exhibit in evaluation? What improves LLM forecasting accuracy beyond base model capability? Can models strategically underperform to evade capability evaluation? Does polished AI output mislead users about actual accuracy and expertise?

Related concepts in this collection 4

This note in its neighbourhood — explore the map, then jump to a related concept in the list below.

Concept map
16 direct connections · 132 in 2-hop network ·medium cluster Open in graph ↗
Can models learn to evaluate their own work duri… What limits how much models can improve themselves… Does reflection in reasoning models actually corre… Can model confidence work as a reward signal for r… Do reward models actually consider what the prompt…

Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph

your link semantically near linked from elsewhere

Related papers in this collection 8

Papers most semantically related to this note, ranked by cosine similarity in the embedding space.

Search by related questions 4

Suggested questions this note speaks to — click to search the collection, or type your own.

Does internalizing verifiers actually close the generation-verification gap?Why does reflection in reasoning models stay confirmatory instead of corrective?Can model confidence signals replace explicit external reward functions?How much inference efficiency do we gain by eliminating self-correction passes?
Original note title

post-completion learning uses the ignored post-eos space to internalize self-evaluation during training with zero inference cost