If you keep feeding an AI facts that contradict what it already believes, does it eventually learn — or stay unpredictable?
Can scaling up contradictory training data overcome unpredictable override effects?
This explores whether piling on more training data that contradicts what a model already believes can make that belief-revision reliable, or whether the unpredictability stays no matter how much data you add.
The closest evidence in the collection says no, and the reason has little to do with volume. Synthetic-document fine-tuning can insert new associations into a model predictably, but when the new documents contradict something the model already holds, the result is unpredictable. That makes the intervention uncontrollable regardless of its strength Can training data edits reliably override what models already believe?. One limit on this: the note is about how hard you push, not a curve of override success against data volume. No note here directly scales up contradictory data and measures the result.
A likely reason is that old knowledge isn't just quiet background that new data drowns out. Language models often produce outputs that ignore their context because associations from training dominate. Text alone can't dislodge a strong prior. The fix the research points to is intervening in the model's internal representations Why do language models ignore information in their context?. A related finding is that when a model faces conflicting signals, removing the misleading cue can make it worse, not better. The task is to integrate the conflicting signals, not to filter one out Why does removing spurious cues sometimes hurt model performance?. If the old association and the new one have to be reconciled inside the model, adding more copies of the new one doesn't obviously settle the conflict.
The RL results suggest that more optimization pressure tends to amplify whatever is already dominant. RL training locks onto one pretraining format within the first epoch and collapses the alternatives, and which one wins depends on model scale, not on which is best Does RL training collapse format diversity in pretrained models?. Training on near-impossible problems teaches degenerate shortcuts that spill into capabilities the model already had Do overly hard RLVR samples actually harm model capabilities?. When reward variance is low, policies fall back into generic templates Why do language models collapse into generic templates?. These are RL settings, not synthetic-document editing, but they point the same way. Scaling up a training signal can reinforce the model's existing tendencies or damage neighbouring ones. It doesn't necessarily resolve a conflict.
What seems to work is changing how the data relates to the model, not how much of it there is. Self-correction fails when trained on offline traces because those errors don't match the model's own, and it succeeds with online RL on the model's actual mistakes Why does self-correction training on offline data fail?. Consistency training uses the model's own clean responses as targets, which avoids the staleness of standard fine-tuning Can models learn to ignore irrelevant prompt changes?. A different route skips the weights and edits frozen hidden representations directly, with far fewer parameters than LoRA Can editing hidden representations beat weight updates for finetuning?.
Scale does help in one nearby case. Walmart's student ranking models beat their LLM teachers once trained on enough teacher-labeled data Can smaller models outperform their LLM teachers with enough data?. That works because more data broadens coverage. It doesn't overwrite anything the model firmly believes. So the pattern in the corpus is that scale helps you add, and overriding needs precision.
Sources 10 notes
Synthetic documents add novel information to models predictably, but contradict and revise existing associations unpredictably. This unpredictability makes such interventions uncontrollable regardless of their strength.
Research demonstrates that LMs generate outputs inconsistent with their context because parametric knowledge from training dominates over in-context information. Textual prompting alone cannot override strong priors; causal intervention in representations is required.
Removing spurious cues degrades performance in heuristic override tasks, opposite to shortcut learning predictions. The failure mode is integrating conflicting signals rather than ignoring distractors—a frame problem, not feature selection.
Controlled experiments show RL consistently amplifies one format distribution from pretraining within the first epoch while collapsing alternatives. The winning format depends on model scale, not necessarily performance, and is largely hidden when starting from proprietary pretrained models.
Training on nearly-impossible problems causes models to learn degenerate shortcuts rather than genuine reasoning, and these shortcuts contaminate pre-existing capabilities. Group-relative normalization treats rare accidental successes as high-advantage trajectories, reinforcing answer repetition and computation-skipping instead of sound reasoning patterns.
Show all 10 sources
When within-prompt reward variance is low, task gradients weaken and regularization dominates, pushing policies toward generic outputs. SNR-Aware Filtering—selecting high-variance prompts before updates—recovers performance across tasks and scales.
SFT on offline correction traces fails because training errors don't match test errors and models collapse into single correction modes. Multi-turn online RL under the model's own error distribution successfully trains self-correction by letting models practice correcting their actual mistakes.
Two methods—BCT (output-level) and ACT (activation-level)—train models to respond identically to clean and wrapped prompts by using the model's own clean responses as targets, eliminating specification and capability staleness inherent in standard SFT.
ReFT learns task-specific interventions on frozen model representations rather than updating weights, with LoReFT (low-rank linear subspace variant) dramatically outperforming LoRA across reasoning, instruction-following, and NLU benchmarks while using far fewer parameters.
Walmart's student cross-encoders outperformed their LLM teachers when trained on sufficiently large augmented datasets of teacher-labeled queries. The student's broader input distribution exposure, smoothed by teacher predictions, enabled better generalization than the teacher achieved.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Echo Chamber: RL Post-training Amplifies Behaviors Learned in Pretraining
- Mechanistically Interpreting the Role of Sample Difficulty in RLVR for LLMs
- Reinforcement Learning for Reasoning in Large Language Models with One Training Example
- Critique-GRPO: Advancing LLM Reasoning with Natural Language and Numerical Feedback
- Shallow Beliefs: Synthetic document finetuning does not inoculate against emergent misalignment from reward hacking
- How new data permeates LLM knowledge and how to dilute it
- Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations?
- Planted in Pretraining, Swayed by Finetuning: A Case Study on the Origins of Cognitive Biases in LLMs