Can you rewrite what an AI already believes by training it on fake documents — and do model size and variety change that?
How do model size and document diversity interact in SDF override success?
This explores whether the size of the model being edited and the variety of the synthetic training documents jointly decide if synthetic document finetuning (SDF, training on generated documents that assert a new fact) can overwrite something a model already believes.
This explores whether the size of the model being edited and the variety of the synthetic training documents jointly decide if synthetic document finetuning (SDF, training on generated documents that assert a new fact) can overwrite something a model already believes. The corpus has no study that varies both together, so it can't tell you how the two interact. It does have a firm result on the override side and several adjacent findings about size and diversity.
The override result is the sturdy part. SDF adds new information to a model predictably, but when the documents contradict an existing association, the outcome is unpredictable, and this holds however strong the intervention is (Can training data edits reliably override what models already believe?). A test of SDF as a fix for reward hacking found the same thing. It didn't reliably override the behavior, and the effect was erratic rather than just weak (Does synthetic document finetuning fail at larger scales?). That paper limits its claim to the scales it tested, so whether bigger models change the picture is an open question, not a measured trend. Since the failure looks like inconsistency rather than a shortfall in dose, 'add more documents' isn't an obvious fix.
Model size shows up twice on the diversity side. Someone has to write the synthetic documents, and the corpus suggests bigger writers may produce a less varied set than you'd expect. Models around 500M parameters generate more distinct outputs per sample than larger ones, because larger models concentrate probability on their preferred outputs (Why aren't bigger models better for generating diverse outputs?). How varied the generator is also depends on how it was tuned. Preference tuning reduces diversity in code and increases it in creative writing (Does preference tuning always reduce diversity the same way?). An SDF corpus is only as varied as the model that wrote it, and that varies with size and tuning.
On the model being edited, the corpus offers only hints, and these are inferences, not tested SDF results. Larger models are more confident and more robust to rephrasing (Does model confidence predict robustness to prompt changes?). If that also holds for beliefs, a bigger model might shrug off many paraphrases of the same counter-claim, so extra diversity wouldn't help much. Separately, in-weight finetuning is bounded by model size and overwrites prior knowledge at the cost of general capability (Can models store unlimited facts without growing larger?). So a larger model has more entrenched knowledge to push against, and forcing an override can damage other abilities.
What's missing is the experiment that would answer your question: cross model size with document diversity and measure how often the override succeeds. Until someone runs it, the corpus says three things. Override fails erratically at the scales tested. Diversity is harder to get than it looks. And bigger models may resist better, though that hasn't been shown for SDF.
Sources 6 notes
Synthetic documents add novel information to models predictably, but contradict and revise existing associations unpredictably. This unpredictability makes such interventions uncontrollable regardless of their strength.
Research shows SDF cannot reliably inoculate models against misalignment from reward hacking, with the effect unpredictable rather than merely weak. The conclusion is explicitly scoped to the specific scales tested, leaving extrapolation uncertain.
Research shows that for synthetic data generation, models around 500M parameters outperform larger ones in output diversity per sample. Larger models concentrate probability mass on preferred outputs, reducing the variety of distinct samples generated within a fixed budget.
RLHF reduces lexical-syntactic diversity in code generation but increases it in creative writing. The direction depends on what each domain incentivizes: code rewards convergence toward correct solutions, while creative writing rewards stylistic distinctiveness.
ProSA found that when models are highly confident, they resist prompt rephrasing; low confidence causes major output swings. Larger models, few-shot examples, and objective tasks all correlate with higher confidence and greater robustness.
Show all 6 sources
A formal proof and experiments show in-weight memorization is bounded by model size, while tool-use enables unbounded factual recall through a simple circuit. In-weight finetuning also degrades general capability by overwriting prior knowledge.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Shallow Beliefs: Synthetic document finetuning does not inoculate against emergent misalignment from reward hacking
- Scaling Synthetic Data Creation with 1,000,000,000 Personas
- Reasoning-Driven Synthetic Data Generation and Evaluation
- Foundation Priors
- Orchestrating Synthetic Data with Reasoning
- Provable Benefits of In-Tool Learning for Large Language Models
- Natural Emergent Misalignment From Reward Hacking In Production RL
- Evaluating the Diversity and Quality of LLM Generated Content