Bigger, better-trained AI models actually get worse at going along with a user's stated belief when it clashes with facts.
Do larger instruction-tuned models actually track user beliefs better?
This explores whether bigger, instruction-tuned models get better at noticing and working with what a user believes, especially when that belief clashes with what the model 'knows' from training.
This explores whether bigger, instruction-tuned models get better at noticing and working with what a user believes, especially when that belief clashes with what the model 'knows' from training. The corpus points the other way. Across 18 models, larger and instruction-tuned versions were *less* likely to go along with a user's stated belief when it contradicted world knowledge Do larger models follow stated beliefs less often?. Scale and assistant training seem to make models lean harder on what they absorbed during training and pay less attention to the frame the user sets up.
Whether that counts as better or worse depends on the job. If you want a model that won't simply agree with a false premise, this looks like progress: less sycophancy. If you want a model that can reason *inside* your assumptions ("suppose I believe X, what follows?"), or that models what you think so it can respond to you rather than to the textbook answer, it's a regression. Following a belief and tracking a belief are different skills. The finding suggests bigger assistants are drifting away from both and toward their own priors.
A related result makes the pattern clearer. When the task is predicting what real people will say, pretrained base models given a few short dialogue samples beat instruction-tuned assistants prompted with personas Do pretrained models simulate humans better than instruction-tuned assistants?. The explanation is task mismatch: training a model to be a helpful assistant teaches it to answer like an assistant, which works against reproducing how a particular human thinks and talks. Another note adds that instruction tuning may teach models mostly what a good *output* looks like rather than deeper understanding of the task Does instruction tuning teach task understanding or output format?. If so, there's little reason to expect it to sharpen a model's grip on anyone's mental state.
This matters beyond benchmarks. Commentators like Benedict Evans argue that LLMs can infer the 'why' behind user behavior well enough for platforms to rent that understanding through an API Can LLMs infer user needs better than owned behavioral data?. The same note reports that LLMs make up 35–49% of their claims about user attributes. A model that confidently describes you while leaning on its priors is producing a plausible guess about you, not tracking you. And when many people rely on the same model, its defaults can pull users toward its framings instead of reflecting theirs Do large language models narrow human expression and thought?.
The corpus here has one direct study and several adjacent ones, not a full theory-of-mind comparison across model sizes. Still, the takeaway is clear: scale plus assistant training buys confidence in the model's own view of the world, not closer attention to yours. If belief tracking matters for your use case, a smaller or base model with good examples may beat the biggest assistant.
Sources 5 notes
Across 18 LLMs tested with EoBench, bigger models and instruction-tuned variants showed lower rates of context-following when users expressed beliefs that contradicted world knowledge. The effect suggests instruction-tuning strengthens reliance on parametric knowledge.
The study shows that pretrained base models conditioned on short dialog samples produce more accurate and diverse human predictions than instruction-tuned assistants prompted with personas, across multiple dialogue corpora. The mechanism is task mismatch: assistant optimization systematically degrades human simulation performance.
Models trained on semantically empty or deliberately incorrect instructions achieve comparable performance to those trained on full correct instructions, achieving 43% vs random baseline 42.6%. The semantic content of instructions appears largely irrelevant; what transfers is knowledge of the output space.
Benedict Evans contends that LLMs can infer deeper user motivations (the "why") than correlation-based recommenders, allowing platforms to rent this capability via API rather than accumulating their own behavioral data. However, research shows LLMs fabricate 35–49% of user attribute claims, undermining confidence in their inferred understanding.
LLMs mirror skewed slices of human experience shaped by training data regularities, and widespread reliance on identical models amplifies convergence. Co-writing studies show users unconsciously adopt model stances and framings.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Are Emergent Abilities in Large Language Models just In-Context Learning?
- It's Not What You Say, It's How You Say It: Evaluating LLM Responses to Expressions of Belief
- Do Models Really Learn to Follow Instructions? An Empirical Study of Instruction Tuning
- Planted in Pretraining, Swayed by Finetuning: A Case Study on the Origins of Cognitive Biases in LLMs
- Fostering Appropriate Reliance on Large Language Models: The Role of Explanations, Sources, and Inconsistencies
- Pretrained Persona Mixture Models and Tandem Models for Human Simulation
- The Homogenizing Effect of Large Language Models on Human Expression and Thought
- A Survey on Post-training of Large Language Models