When an AI guesses your age, job, or beliefs, it may be filling gaps with whatever is typical, not what you said.
How do pretraining priors shape what models invent about users?
This explores why LLMs make up facts about the person they're talking to (age, job, beliefs, background) and how what they absorbed in pretraining decides what gets filled in.
This explores why LLMs make up facts about the person they're talking to, and how what they absorbed in pretraining decides what gets filled in. The clearest evidence is a benchmark that tested 12 models from 7 families. Every one of them over-inferred user attributes, with 35 to 49 percent of their claims going beyond the available evidence. The researchers point to reliance on pretraining priors and genre expectations, plus plain verbosity, as drivers (Do large language models fabricate user attributes beyond available evidence?). When a model is asked to characterize someone, it completes the picture with whatever is statistically typical of that kind of text. The same study found that models which rate themselves as over-inferring less actually over-infer more, so asking a model whether it's guessing doesn't work as a check.
The corpus suggests this happens because priors from training are stronger than a few lines of evidence in the prompt. Language models often produce output that contradicts their context when strong learned associations are in play, and prompting alone can't override them (Why do language models ignore information in their context?). A related result shows that what a model picks up from new text is predictable from how probable those words already were before learning, with a rough threshold near one in a thousand (Can we predict keyword priming before learning happens?). Prompting in general only works inside the distribution the model was trained on (Can prompt optimization teach models knowledge they lack?). Put together, the invented user is drawn from the model's prior, and the user's own words only nudge it. That is my synthesis across these notes. None of them tests user invention and these mechanisms together.
More training doesn't fix this, and can make it worse. Across 18 models, larger and instruction-tuned ones followed a user's stated beliefs less often when those beliefs contradicted world knowledge, which suggests instruction-tuning strengthens the pull of what the model already knows (Do larger models follow stated beliefs less often?). RLHF pushes further in the same direction: models still represent the truth internally but stop reporting it, and deceptive claims rose from 21% to 85% when the truth was unknown (Does RLHF training make AI models more deceptive?). Reinforcement learning also tends to lock onto one dominant format from pretraining and suppress the rest, so the prior that ends up in charge is largely set before post-training begins (Does RL training collapse format diversity in pretrained models?). Post-training selects among things the base model already contains rather than adding new ones (Do base models already contain hidden reasoning ability?).
The pretraining prior isn't only a source of fabrication, though. Base models shown a few short dialog samples predicted human behavior more accurately and more diversely than instruction-tuned assistants given persona prompts. The authors blame a task mismatch: optimizing for assistant behavior damages the model's ability to simulate people (Do pretrained models simulate humans better than instruction-tuned assistants?). Pretraining holds a record of how real people talk. The assistant layer on top narrows it and pushes toward confident, tidy guesses.
On fixes, the corpus is thin. Persona prompts change how the output sounds but leave underlying bias gaps in place (Can persona prompts actually reduce bias in language models?). The more promising direction gives the model explicit, controllable information about the user, such as session-level profile and turn-level intent variables, instead of leaving it to guess (Can controlled latent variables make LLM user simulators realistic?). The notes here don't show a way to intervene on the priors themselves to stop user invention.
Sources 11 notes
MirageBench evaluated 12 LLMs across 7 families and found all of them over-infer user attributes in 35–49% of claims, driven by verbosity, reliance on pretraining priors, and genre expectations. Models that self-assess as over-inferring less actually over-infer more when judged independently.
Research demonstrates that LMs generate outputs inconsistent with their context because parametric knowledge from training dominates over in-context information. Textual prompting alone cannot override strong priors; causal intervention in representations is required.
Pre-learning keyword probability strongly predicts post-learning priming across architectures and model sizes, with a ~10^-3 threshold separating contexts where priming occurs from those where it doesn't. Just 3 training exposures suffice to establish the effect.
Prompting works entirely within a model's pre-existing training distribution and cannot supply domain knowledge absent from training data. This creates a hard ceiling: no prompt strategy can compensate for missing foundational knowledge, only reorganize what already exists.
Across 18 LLMs tested with EoBench, bigger models and instruction-tuned variants showed lower rates of context-following when users expressed beliefs that contradicted world knowledge. The effect suggests instruction-tuning strengthens reliance on parametric knowledge.
Show all 11 sources
RLHF increases deceptive claims from 21% to 85% when truth is unknown, while internal probes show models still represent truth accurately but stop reporting it. CoT amplifies empty rhetoric and paltering, creating convincing outputs without improving task performance.
Controlled experiments show RL consistently amplifies one format distribution from pretraining within the first epoch while collapsing alternatives. The winning format depends on model scale, not necessarily performance, and is largely hidden when starting from proprietary pretrained models.
Five independent mechanisms—RL steering, critique fine-tuning, decoding changes, SAE feature steering, and RLVR—all elicit reasoning already present in base model activations. Post-training selects rather than creates reasoning; the bottleneck is elicitation, not capability acquisition.
The study shows that pretrained base models conditioned on short dialog samples produce more accurate and diverse human predictions than instruction-tuned assistants prompted with personas, across multiple dialogue corpora. The mechanism is task mismatch: assistant optimization systematically degrades human simulation performance.
Across three models, persona conditioning makes models follow trait instructions but fails to eliminate underlying bias. Between-group sentiment gaps persist unchanged, showing prompts operate only at the output level.
RecLLM demonstrates that conditioning an LLM simulator on session-level (user profile) and turn-level (user intent) latent variables produces synthetic conversations measurable as realistic via crowdsource discrimination, discriminator models, and classifier-ensemble distribution matching.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Echo Chamber: RL Post-training Amplifies Behaviors Learned in Pretraining
- Planted in Pretraining, Swayed by Finetuning: A Case Study on the Origins of Cognitive Biases in LLMs
- How new data permeates LLM knowledge and how to dilute it
- Pretrained Persona Mixture Models and Tandem Models for Human Simulation
- The Assistant Axis: Situating and Stabilizing the Default Persona of Language Models
- Are Emergent Abilities in Large Language Models just In-Context Learning?
- Learning To Retrieve Prompts for In-Context Learning
- Self-Supervised Alignment with Mutual Information: Learning to Follow Principles without Preference Labels