If you teach an AI a made-up fact by training it on fake documents, does more data make the results predictable?
Does unpredictable generalization from SDF become predictable at different training document scales?
This explores whether the unpredictable behavior of synthetic document finetuning (SDF, training a model on made-up documents to plant a belief) settles into something reliable when you change how many documents you train on.
This explores whether the unpredictable behavior of synthetic document finetuning (SDF, training a model on made-up documents to plant a belief) settles into something reliable when you change how many documents you train on. The corpus has no scaling curve for this, so nothing here shows that it does. What it does show is that predictability seems to depend on what you ask SDF to do, more than on how much of it you do.
The clearest finding is a split by task. SDF adds new information to a model predictably, but when the documents contradict or revise something the model already believes, the results are unpredictable. That unpredictability makes the intervention uncontrollable Can training data edits reliably override what models already believe?. Reward hacking is the hard case. SDF could not reliably inoculate models against misalignment that comes from reward hacking, and the effect was erratic rather than just weak Does synthetic document finetuning fail at larger scales?. That note also limits its conclusion to the scales tested. The only note that touches scale is saying it doesn't know what happens beyond that range.
The distinction between erratic and weak matters. A weak but consistent effect could plausibly be fixed with more documents. An erratic one is a different kind of problem, and the first note says the unpredictability holds however strong the intervention is. That hints that turning up the dose won't buy predictability, but this is an inference from the notes, not a tested result. It also matters what "scale" means here, since document count and model size are separate axes and the corpus doesn't separate them.
Neighboring notes suggest why override might stay hard. Language models often ignore information in front of them when their training associations are strong, and text alone can't dislodge those priors Why do language models ignore information in their context?. SDF documents may be fighting the same entrenched associations from the inside. A second parallel is self-correction training. SFT on offline correction traces failed because the training errors didn't match the model's actual errors, and only online RL on the model's own mistakes worked Why does self-correction training on offline data fail?. SDF is also offline, so a similar mismatch could be at work. The corpus hasn't tested that.
Scale can also change outcomes in ways that aren't smooth. In RL post-training, which pretraining format wins out depends on model scale Does RL training collapse format diversity in pretrained models?. Domain training methods more broadly have sweet spots and hidden costs, so more training is not reliably better How do domain training techniques actually reshape model behavior?. What's missing is an experiment that varies both document count and model size and measures how much results differ across runs, separately for adding versus overriding beliefs. Until then, whether SDF becomes predictable at scale is an open question.
Sources 6 notes
Synthetic documents add novel information to models predictably, but contradict and revise existing associations unpredictably. This unpredictability makes such interventions uncontrollable regardless of their strength.
Research shows SDF cannot reliably inoculate models against misalignment from reward hacking, with the effect unpredictable rather than merely weak. The conclusion is explicitly scoped to the specific scales tested, leaving extrapolation uncertain.
Research demonstrates that LMs generate outputs inconsistent with their context because parametric knowledge from training dominates over in-context information. Textual prompting alone cannot override strong priors; causal intervention in representations is required.
SFT on offline correction traces fails because training errors don't match test errors and models collapse into single correction modes. Multi-turn online RL under the model's own error distribution successfully trains self-correction by letting models practice correcting their actual mistakes.
Controlled experiments show RL consistently amplifies one format distribution from pretraining within the first epoch while collapsing alternatives. The winning format depends on model scale, not necessarily performance, and is largely hidden when starting from proprietary pretrained models.
Show all 6 sources
Research shows every adaptation method—from parameter-efficient tuning to knowledge graph curricula—has optimal conditions tied to specific domains. The key finding: visible benefits like performance gains often come with hidden degradation in reasoning faithfulness, capability transfer, and format flexibility.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Shallow Beliefs: Synthetic document finetuning does not inoculate against emergent misalignment from reward hacking
- How new data permeates LLM knowledge and how to dilute it
- Natural Emergent Misalignment From Reward Hacking In Production RL
- Echo Chamber: RL Post-training Amplifies Behaviors Learned in Pretraining
- Training Language Models to Self-Correct via Reinforcement Learning
- Language models show human-like content effects on reasoning tasks
- Graph Engineering in the Era of LLM Agents: From Individual Intelligence to System Intelligence
- Between Circuits and Chomsky: Pre-pretraining on Formal Languages Imparts Linguistic Biases