When your code fails and the error message comes back, can the AI itself work out which exact steps caused it?
Can environment feedback alone provide dense credit without a teacher?
This explores whether the feedback an environment sends back (compiler errors, test output, a critique in words) can tell a model which specific tokens or steps were to blame, without a stronger teacher model or a trained process reward model grading each step.
This explores whether the feedback an environment sends back (compiler errors, test output, a critique in words) can tell a model which specific tokens or steps were to blame, without a stronger teacher model or a separately trained step-by-step grader. The corpus says yes, with a twist: the teacher doesn't disappear, it moves inside the model. A method called SDPO shows how. You show the policy its own failed attempt together with the environment's feedback, in-context. That feedback-conditioned version of the model then acts as its own teacher, and its token-by-token judgments work as a process reward model, so no external reward signal is needed Can environment feedback replace scalar rewards in policy learning?.
The reason this works is that feedback carries two different things: how well an action did (evaluative) and how it should change (directive). A scalar reward keeps the first and throws away the second, while token-level distillation can recover it Can scalar rewards capture all the information in agent feedback?. So 'environment feedback alone' only works if the feedback is rich text, not a pass/fail bit. The dense credit was in the message all along, and collapsing it to a single number was the bottleneck.
When the environment says little, the corpus offers other ways to get dense credit without a teacher. ΔBelief-RL credits each turn by how much the agent's own confidence in the right answer shifts, with no critic and no process reward model. Small models trained this way on 20 Questions matched or beat larger baselines Can an agent's own beliefs guide credit assignment without critics?. Other approaches manufacture the signal internally. SERL has the model alternate between answering and judging, and rewards the consistency of its own rulings, lifting its AlpacaEval win rate from 52.37% to 59.90% Can models learn to judge themselves without external rewards?. Post-completion learning trains the model to compute its own reward in the unused space after the end-of-sequence token, at zero inference cost Can models learn to evaluate their own work during training?. These signals are coarser than SDPO's token-level ones, and they come from the model's own judgment rather than from the environment.
A self-teacher still inherits some of the problems of an outside one. Teachers that see correct answers and verifier output produce confident, short traces, and students copy that style. It helps in-domain and hurts out-of-distribution problems that call for caution Does richer teacher context hurt student generalization?. A feedback-conditioned self-teacher is in a similar position, so this is a risk to check. The corpus doesn't test it on SDPO directly. The upside is that a self-teacher matches the student exactly, which avoids the compatibility problem where teacher refinements beyond the student's learning frontier make things worse Does teacher-refined data always improve student model performance?. Distillation evidence also suggests that this kind of signal steers a model toward correct paths it can already reach, and doesn't raise its ceiling. Signal quality matters more than teacher size Does on-policy distillation actually expand student capability?. Dense rewards can also be gamed, and one fix is to use rubrics as accept-or-reject gates while token-level rewards optimize within valid answers Can rubrics and dense rewards work together without hacking?.
The corpus has no head-to-head comparison of a self-teacher against an external one, and none showing how far feedback-only credit degrades when the feedback is thin or noisy. For now, the evidence supports 'yes, if the feedback is rich enough and the model can re-read its own mistakes.'
Sources 9 notes
SDPO converts tokenized environment feedback into dense gradient signals by using the feedback-conditioned policy as a self-teacher. The policy, when given retrospective evidence of its mistakes in-context, implicitly acts as its own process reward model, making external reward signals unnecessary.
Natural feedback carries two orthogonal types of information: evaluative (how well an action performed) and directive (how it should change). Scalar rewards capture evaluation but discard directional specifics that token-level distillation can recover, making the two complementary rather than redundant.
ΔBelief-RL uses log-ratios of sequential probability estimates to assign per-turn credit without critic networks or process reward models. Tested on 20 Questions, smaller models trained this way matched or exceeded prior SOTA and larger baselines while generalizing beyond training.
SERL enables self-improving language models by having them alternate between generating responses and judging them pairwise, deriving rewards from ranking consistency and self-consistency of judgments. On AlpacaEval, this reached 59.90% win rate without external signals, up from 52.37%.
Post-Completion Learning exploits unused sequence space after model output to train self-assessment capabilities during training while maintaining zero inference cost. The model learns to compute its own reward functions, internalizing evaluation rather than relying on external reward models.
Show all 9 sources
Teachers conditioned on correct answers and verifier output produce confident, concise traces that students inherit. This style suppresses uncertainty expression, optimizing in-domain performance while degrading generalization to out-of-distribution problems that require epistemic caution.
Teacher-refined data degrades performance when it exceeds the student's learning frontier, even if objectively higher quality. Students should filter refinements using their own statistical profile to retain only compatible improvements.
On-policy distillation steers students toward correct reasoning paths within their existing capability envelope rather than raising the ceiling. Signal quality and diversity matter far more than teacher scale; a smaller teacher with high-fidelity guidance outperforms larger teachers without it.
DRO shows that using rubrics to accept or reject rollout groups—rather than converting rubric scores into dense rewards—prevents reward hacking. This separation preserves the categorical strength of rubrics while letting token-level rewards optimize within valid answers.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Reinforcement Learning via Self-Distillation
- Reward Reasoning Model
- Does On-Policy Distillation Really Distill? From Noisy Teacher to Self-Improvement
- AI4AI at Test-Time: Strong-to-Weak Capability Transfer via Harnesses
- Supervised Reinforcement Learning: From Expert Trajectories to Step-wise Reasoning
- Temporal Self-Rewarding Language Models: Decoupling Chosen-Rejected via Past-Future
- Intrinsic Credit Assignment for Long Horizon Interaction
- Meta-Rewarding Language Models: Self-Improving Alignment with LLM-as-a-Meta-Judge