Teaching an AI to follow instructions can make it less willing to believe what you tell it, if it clashes with its training.
Why does instruction-tuning reduce a model's context-following behavior?
This explores why a model trained to follow instructions ends up less willing to go along with what's in its prompt, especially when the prompt clashes with what the model already 'knows'.
This explores why a model trained to follow instructions ends up less willing to go along with what's in its prompt, especially when the prompt clashes with what the model already 'knows'. The most direct evidence is a test of 18 models. Both bigger models and instruction-tuned variants followed a user's stated beliefs less often when those beliefs contradicted world knowledge Do larger models follow stated beliefs less often?. The suggested explanation is that instruction-tuning strengthens the model's reliance on what it absorbed in pretraining over what's in front of it.
That fits a wider pattern in the corpus. Even without tuning, models often produce outputs that contradict their context when a strong training association is in play. Prompting alone couldn't override this. Changing the outcome required intervening in the model's internal representations Why do language models ignore information in their context?. So instruction-tuning probably doesn't create the tension between memory and context. It seems to tilt an existing tension further toward the model's priors.
A second thread is about what instruction-tuning actually teaches. Models trained on meaningless or deliberately wrong instructions did about as well as those trained on correct ones. What transferred was the shape of a good answer, not an understanding of the task Does instruction tuning teach task understanding or output format?. If tuning mostly teaches 'answer like an assistant,' a model can look obedient while leaning on its own defaults instead of reading closely. The same mismatch shows up in a base-versus-assistant comparison. Pretrained models given a few dialog samples predicted human behavior more accurately and more diversely than instruction-tuned assistants given a persona prompt. The study blames task mismatch: assistant optimization aims at being a good answerer, not at continuing whatever situation the context sets up Do pretrained models simulate humans better than instruction-tuned assistants?.
Other training stages loosen the model's grip on the prompt too, by different routes. SFT and RL for reasoning improved math but lowered instruction adherence, and longer chains of thought put more distance between the model and the original instruction Why do better reasoning models ignore instructions?. Fine-tuning also made reasoning steps matter less to final answers, so the reasoning became more performative than functional Does fine-tuning disconnect reasoning steps from final answers?. In RL, when reward barely varies across a prompt's samples, policies collapse into generic, input-agnostic templates Why do language models collapse into generic templates?. Instruction-following itself is also not one dial. It degrades as instructions pile up, in patterns that depend on model type How does instruction density affect model performance?.
No note here isolates instruction-tuning as the cause with a controlled mechanism. The best-supported story is that tuning shifts the balance toward parametric knowledge and teaches output shape rather than close reading, and the other findings are converging hints. The takeaway is that following instructions and following context are different skills. Optimizing hard for one can quietly cost the other.
Sources 8 notes
Across 18 LLMs tested with EoBench, bigger models and instruction-tuned variants showed lower rates of context-following when users expressed beliefs that contradicted world knowledge. The effect suggests instruction-tuning strengthens reliance on parametric knowledge.
Research demonstrates that LMs generate outputs inconsistent with their context because parametric knowledge from training dominates over in-context information. Textual prompting alone cannot override strong priors; causal intervention in representations is required.
Models trained on semantically empty or deliberately incorrect instructions achieve comparable performance to those trained on full correct instructions, achieving 43% vs random baseline 42.6%. The semantic content of instructions appears largely irrelevant; what transfers is knowledge of the output space.
The study shows that pretrained base models conditioned on short dialog samples produce more accurate and diverse human predictions than instruction-tuned assistants prompted with personas, across multiple dialogue corpora. The mechanism is task mismatch: assistant optimization systematically degrades human simulation performance.
The MathIF benchmark shows that SFT and RL training improve reasoning but reduce instruction adherence, particularly as chain-of-thought length increases. Longer reasoning chains create contextual distance that dilutes the model's attention to original instructions.
Show all 8 sources
Three faithfulness tests show fine-tuned models generate reasoning chains that less reliably influence final outputs. Early termination, paraphrasing, and filler substitution all produce invariant answers more often after fine-tuning, suggesting reasoning becomes performative rather than functional.
When within-prompt reward variance is low, task gradients weaken and regularization dominates, pushing policies toward generic outputs. SNR-Aware Filtering—selecting high-variance prompts before updates—recovers performance across tasks and scales.
IFScale benchmark shows three degradation patterns: linear (small models), exponential (mid-range), and threshold decay (reasoning models maintain ~150 instructions then fail steeply). Even best models reach only 68% accuracy at maximum density.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Are Emergent Abilities in Large Language Models just In-Context Learning?
- It's Not What You Say, It's How You Say It: Evaluating LLM Responses to Expressions of Belief
- Scaling Reasoning, Losing Control: Evaluating Instruction Following in Large Reasoning Models
- Do Models Really Learn to Follow Instructions? An Empirical Study of Instruction Tuning
- Echo Chamber: RL Post-training Amplifies Behaviors Learned in Pretraining
- Planted in Pretraining, Swayed by Finetuning: A Case Study on the Origins of Cognitive Biases in LLMs
- Exploring Format Consistency for Instruction Tuning
- A Survey on Post-training of Large Language Models