When an AI is trained on scores, why does a firm 'never do this' rule become just a price to pay?
Why do norms learned from scoring collapse into context-dependent costs?
This explores why training a model on scores (reward models, annotator ratings) tends to turn a firm norm like 'don't do this' into a situational price, where the behavior is just costly in some contexts and fine in others.
This explores why training on scores tends to turn a firm norm into a situational price: 'don't do this' becomes 'doing this loses points here.' The corpus has no note that states this directly, so what follows is a reading assembled from adjacent findings. It is not a settled result.
The first reason is that a score keeps only half of what feedback says. Natural feedback carries an evaluative part (how well did that go) and a directive part (what should change) Can scalar rewards capture all the information in agent feedback?. A scalar reward keeps the first and throws away the second. A norm is mostly directive, since it says what to do instead. Once it has been squeezed into a number, all that is left is a penalty attached to a situation, and a penalty can always be outweighed by other terms in that situation.
The second reason is that the scores were never clean measurements of norms. Annotation responses mix genuine preferences, non-attitudes, and preferences that annotators construct on the spot, and these can only be told apart by checking whether the answer holds across measurement conditions Do all annotation responses measure the same underlying thing?. A held norm and a judgment improvised for one prompt land in the training data as the same kind of number. A constructed preference is context-dependent by definition, because it is built from the framing in front of the annotator. When the reward model averages over all three, the stable part and the improvised part blur together, and what the model learns looks like a cost that moves with context.
Scoring also only teaches what varies. When rewards barely differ across responses to the same prompt, the task gradient weakens and the model drifts toward generic, input-agnostic outputs Why do language models collapse into generic templates?. The same logic suggests that a norm which is nearly always followed in the data produces almost no learning signal. Only the contested cases, where the norm trades off against something else, generate gradient. The model therefore learns the norm mostly from the situations where it is negotiable. A related finding is that strong priors and current context compete instead of one acting as a hard rule Why do language models ignore information in their context?, which fits the picture of norms as weights rather than constraints.
The corpus does point to ways of keeping a norm as a norm. Prompting a model to err, reflect, and write down the principle behind each mistake improves reasoning without any new labels Does learning from mistakes improve in-context learning?. Treating failed episodes as abstracted lessons and successes as concrete demonstrations also outperforms uniform handling Should successful and failed episodes be processed differently?. Both keep the directive content as explicit statements instead of reducing it to a number. The question this leaves open is whether a norm can be kept firm without ever being written down as a rule, and nothing here answers that.
Sources 6 notes
Natural feedback carries two orthogonal types of information: evaluative (how well an action performed) and directive (how it should change). Scalar rewards capture evaluation but discard directional specifics that token-level distillation can recover, making the two complementary rather than redundant.
Behavioral science reveals that annotations contain genuine preferences, non-attitudes, and constructed preferences—distinguishable by consistency across measurement conditions. Treating them uniformly contaminates reward model training and downstream alignment.
When within-prompt reward variance is low, task gradients weaken and regularization dominates, pushing policies toward generic outputs. SNR-Aware Filtering—selecting high-variance prompts before updates—recovers performance across tasks and scales.
Research demonstrates that LMs generate outputs inconsistent with their context because parametric knowledge from training dominates over in-context information. Textual prompting alone cannot override strong priors; causal intervention in representations is required.
LEAP demonstrates that models achieve better performance on reasoning and math tasks by intentionally erring on few-shot examples, reflecting on mistakes, and deriving explicit task-specific principles—without additional labeled data or fine-tuning.
Show all 6 sources
SkillRL demonstrates that treating successful episodes as concrete demonstrations and failures as abstracted lessons achieves state-of-the-art performance on complex tasks while using substantially less context than uniform approaches. The asymmetry mirrors human expert reasoning and avoids the degradation seen in uniform consolidation methods.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Mechanistically Interpreting the Role of Sample Difficulty in RLVR for LLMs
- Learning To Retrieve Prompts for In-Context Learning
- Measuring Human Preferences in RLHF is a Social Science Problem
- In-Context Principle Learning from Mistakes
- RAGEN-2: Reasoning Collapse in Agentic RL
- How new data permeates LLM knowledge and how to dilute it
- Reinforcement Learning Finetunes Small Subnetworks in Large Language Models
- Language models show human-like content effects on reasoning tasks