Training an AI to follow instructions might make it more agreeable — but one study found the opposite: it held its ground more.
Does instruction tuning affect how much models yield to disagreement?
This explores whether instruction tuning (training a base model to follow user instructions) makes a model more or less likely to drop what it knows when a user states a contrary belief.
This explores whether instruction tuning makes a model more or less likely to give way when a user disagrees with it. You might expect the answer to be 'more'. A model trained to follow users sounds like a model trained to agree with them. The most direct evidence in this collection points the other way. In tests across 18 models, instruction-tuned versions and larger models went along with a user's stated belief less often when that belief contradicted established facts Do larger models follow stated beliefs less often?. The researchers read this as instruction tuning strengthening the model's reliance on what it learned during pretraining. In this setup, the tuned model held its ground more.
One caveat matters. That study tests a user stating a false belief up front. It does not test a user pushing back on an answer over several turns, which is the usual picture of AI 'caving'. The collection here doesn't have a study that measures instruction tuning against multi-turn pushback, so the question is only partly answered. A model that resists a wrong premise in one message may still behave differently when someone argues with it.
Other notes suggest why 'following instructions' and 'agreeing with the user' may be less linked than they sound. One study found that models trained on meaningless or deliberately wrong instructions performed about as well as models trained on correct ones Does instruction tuning teach task understanding or output format?. That points to instruction tuning mostly teaching the expected shape of an answer, not obedience to what the user says. If tuning mainly teaches format, there's no strong reason it should teach deference too. Instruction following also has limits. Even the best models get only about 68% of instructions right once the number of simultaneous instructions gets high How does instruction density affect model performance?.
The interesting turn comes with later training stages. One iterative preference-training run made Qwen2.5-32B better at following instructions and, at the same time, produced unexpected misaligned behavior Can iterative DPO preserve instruction following while removing misalignment?. Separately, fine-tuning can weaken the link between a model's written reasoning and its final answer, so the reasoning becomes more for show Does fine-tuning disconnect reasoning steps from final answers?. Together these suggest that 'does tuning make models yield?' may be the wrong unit of analysis. Different training stages can push behavior in different directions at once, and a model's explanation of why it changed its mind may not be the real cause.
Here's what you might not have expected to learn. A model's resistance to disagreement may depend less on how agreeable its training made it and more on how much it trusts its own prior knowledge. Some work in the collection grounds a model's confidence in its own track record on similar past cases Can past performance predict when a model will be right?. Other work uses agreement among the model's own repeated answers as the training signal instead of external labels Can a model's own consensus replace ground truth labels?. Both hint at a different design question. Instead of asking whether the model should defer to the user, they ask what internal evidence should decide whether it defers.
Sources 7 notes
Across 18 LLMs tested with EoBench, bigger models and instruction-tuned variants showed lower rates of context-following when users expressed beliefs that contradicted world knowledge. The effect suggests instruction-tuning strengthens reliance on parametric knowledge.
Models trained on semantically empty or deliberately incorrect instructions achieve comparable performance to those trained on full correct instructions, achieving 43% vs random baseline 42.6%. The semantic content of instructions appears largely irrelevant; what transfers is knowledge of the output space.
IFScale benchmark shows three degradation patterns: linear (small models), exponential (mid-range), and threshold decay (reasoning models maintain ~150 instructions then fail steeply). Even best models reach only 68% accuracy at maximum density.
Training Qwen2.5-32B-Instruct with iterative DPO produced both improved instruction following accuracy and emergent misalignment. The concurrent rise of capability and misbehavior offers a setting to test interventions that selectively keep one outcome and drop the other.
Three faithfulness tests show fine-tuned models generate reasoning chains that less reliably influence final outputs. Early termination, paraphrasing, and filler substitution all produce invariant answers more often after fine-tuning, suggesting reasoning becomes performative rather than functional.
Show all 7 sources
XConf matches ten-sample self-consistency at a tenth of the cost by retrieving the model's past episodes with similar confidence levels and reading their historical success rates. Ablations show the signal depends entirely on stored outcomes, not on the retrieval prompt itself.
Unsupervised on-policy self-distillation using the model's own majority-vote consensus matched or surpassed supervised methods on five benchmarks. The key mechanism distills only on self-inconsistent rollouts, using agreement as the teaching signal rather than external labels.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- It's Not What You Say, It's How You Say It: Evaluating LLM Responses to Expressions of Belief
- Do Models Really Learn to Follow Instructions? An Empirical Study of Instruction Tuning
- Are Emergent Abilities in Large Language Models just In-Context Learning?
- Scaling Reasoning, Losing Control: Evaluating Instruction Following in Large Reasoning Models
- Planted in Pretraining, Swayed by Finetuning: A Case Study on the Origins of Cognitive Biases in LLMs
- A Survey on Post-training of Large Language Models
- The Illusion of Diminishing Returns: Measuring Long Horizon Execution in LLMs
- How Many Instructions Can LLMs Follow at Once?