When an AI can't tell which of two answers is better, what keeps it learning from feedback?
How does temporal anchoring maintain learning signals when preference gaps collapse?
This explores what keeps a training signal alive over time when the gap between a 'better' and a 'worse' response shrinks toward nothing, and whether tying learning to a fixed reference point (a stored episode, a base model, an outcome) can substitute for a sharp preference gap.
This explores what keeps a training signal alive over time when the gap between a 'better' and a 'worse' response shrinks toward nothing, and whether tying learning to a fixed reference point can substitute for a sharp preference gap. The corpus has no note on 'temporal anchoring' by that name, so what follows is a reading across neighboring ideas, not a direct answer. The neighbors do agree on one thing: when a gap collapses, the fix is usually to move the signal somewhere else, not to sharpen the gap.
First, ask whether the gap was ever real. Annotation responses turn out to be a mix of genuine preferences, non-attitudes, and preferences constructed on the spot by the question itself, and you can tell them apart by whether they stay consistent across measurement conditions Do all annotation responses measure the same underlying thing?. A gap that 'collapses' during training may be the consistent part surviving while the noisy part washes out. Treating all three as one signal is what contaminates the reward model in the first place.
Second, several notes anchor learning to something fixed so no pairwise gap is needed. Consistency training uses the model's own clean responses as the target, and teaches it to answer the same way when the prompt is wrapped in irrelevant changes Can models learn to ignore irrelevant prompt changes?. SAMI skips preference labels entirely and maximizes the mutual information between a written set of principles and the model's responses, so the principles are the anchor Can models learn behavioral principles without preference labels?. Staying close to the base model does a similar job over time: low KL drift preserves plasticity, while models that wander far stall when the task changes Does staying close to the base model preserve learning ability?. Proxy-tuning takes this to the limit by leaving the weights untouched and shifting behavior at decoding time Can decoding-time tuning preserve knowledge better than weight fine-tuning?.
Third, the 'temporal' part shows up as sequence and outcome. In-context learning of sequential decisions needs whole trajectories from the same environment, because the signal lives in the ordered run of steps and not in isolated examples Why do trajectories matter more than individual examples for in-context learning?. Reflexion anchors learning to a clear success-or-failure result at the end of an episode. That binary outcome keeps the agent from rationalizing, and the agent's written reflections carry the lesson forward without any weight update Can agents learn from failure without updating their weights?. SkillRL goes further and processes wins and losses differently, keeping successes as concrete demonstrations and failures as abstracted lessons Should successful and failed episodes be processed differently?.
Finally, two notes explain why a signal can look dead when it isn't. RL training moves through phases, from getting execution right to getting strategy right, so a signal that has gone flat in one phase can come alive in the next Does RL training follow a predictable two-phase learning sequence?. And when numerical rewards plateau, natural-language critiques can still push a stuck model to correct answers, because a number can't say why something failed Can natural language feedback overcome numerical reward plateaus?. If a specific paper on temporal anchoring exists in the library, it didn't surface in these results.
Sources 10 notes
Behavioral science reveals that annotations contain genuine preferences, non-attitudes, and constructed preferences—distinguishable by consistency across measurement conditions. Treating them uniformly contaminates reward model training and downstream alignment.
Two methods—BCT (output-level) and ACT (activation-level)—train models to respond identically to clean and wrapped prompts by using the model's own clean responses as targets, eliminating specification and capability staleness inherent in standard SFT.
SAMI finetunes language models to increase mutual information between constitutions and responses without preference labels or demonstrations. A mistral-7b trained this way outperformed base and instruction-tuned baselines, and surprisingly, a weaker model could write principles to align a stronger one.
FST-trained models stay up to 70% closer to their base distribution than parameter-only RL, and this reduced drift preserves the model's ability to learn subsequent tasks effectively. Parameter-only approaches stall when task domains change, while low KL drift enables sustained adaptation.
Proxy-tuning closes 88-91% of the alignment gap while surpassing direct fine-tuning on knowledge tasks by leaving base model weights untouched. Direct fine-tuning corrupts knowledge storage in lower layers, whereas proxy-tuning applies distributional shifts that primarily affect reasoning and style.
Show all 10 sources
In-context learning for sequential decision-making requires full or partial trajectories from the same environment level, not just isolated examples. This structural property—trajectory burstiness—allows models to generalize across vastly different tasks without weight updates.
Reflexion demonstrates that unambiguous environmental feedback (success/failure) enables agents to write useful self-diagnoses and improve across episodes without parameter updates. The binary signal prevents rationalization, and keeping reflections uncompressed preserves their usability.
SkillRL demonstrates that treating successful episodes as concrete demonstrations and failures as abstracted lessons achieves state-of-the-art performance on complex tasks while using substantially less context than uniform approaches. The asymmetry mirrors human expert reasoning and avoids the degradation seen in uniform consolidation methods.
Across eight models, RL training consistently shows a first phase where execution correctness drives learning, followed by a second phase where strategic planning becomes the bottleneck. Planning token entropy increases while execution entropy stabilizes, and concentration of optimization on planning tokens yields significant performance gains.
Critique-GRPO shows that models stuck on performance plateaus can generate correct solutions when given chain-of-thought critiques, revealing that numerical rewards lack critical information about why failures occur and how to improve.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- AgentFly: Fine-tuning LLM Agents without Fine-tuning LLMs
- Useful Memories Become Faulty When Continuously Updated by LLMs
- Teaching Large Language Models to Reason with Reinforcement Learning
- Post-training makes large language models less human-like
- Self-distillation Enables Continual Learning
- Spurious Forgetting in Continual Learning of Language Models
- Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustment
- Beyond Preferences in AI Alignment