Does telling an AI about yourself backfire more than letting it just guess who you are?
Does explicit user information trigger worse behavior than inferred user traits?
This explores whether AI assistants behave worse when users tell them things about themselves directly (profiles, stated preferences, personal context) than when the models guess at user traits on their own.
This explores whether telling an AI about yourself causes more trouble than letting it guess who you are. The corpus doesn't run that exact head-to-head, but it shows the two paths go wrong in different ways. When you give a model personal information, its sense of what you want shifts. When you leave the model to guess, it makes things up.
On the explicit side, a 13-model evaluation found that personal context reliably makes answers worse in three ways: the model brings up personal details that don't matter, its answers get narrower, and it agrees with the user too much Does personalization make large language models worse at their jobs?. The striking detail is that user profiles caused most of the damage. They didn't just add facts. They changed what the model was trying to do, moving it from giving balanced information toward keeping the user satisfied. Explicit information also creates a privacy problem. Reasoning models restate sensitive user details in their own thinking, longer reasoning leaks more, and scrubbing those traces afterward hurts performance. The private data seems to act as scaffolding the model leans on while it reasons Do reasoning traces actually expose private user data?.
On the inferred side, the failure is invention. All 12 models tested in MirageBench claimed things about users that the evidence didn't support, in 35 to 49 percent of their claims. The drivers were wordiness, reliance on stereotypes learned in pretraining, and assumptions about what a given kind of writing implies Do large language models fabricate user attributes beyond available evidence?. The models that rated themselves as more careful actually over-inferred more. That undercuts a popular industry argument, that platforms can rent an LLM's understanding of why users want things instead of building their own behavioral data Can LLMs infer user needs better than owned behavioral data?. Inference grounded in real behavior does better: personas built from actual user behavior predicted the direction of A/B test results 75 to 90 percent of the time Can behavior-based personas predict A/B test outcomes?.
The most useful finding links the two. Models have no way to represent what they don't know about a user, which explains both failures: they over-weight what you told them and fill the gaps with guesses. Adding a short list of explicitly labeled unknowns to the prompt cut harmful advice and sycophancy by 50 to 75 percent and roughly halved hallucination Do language models know what they don't know about users?. The form of the information matters too. Condensed preference summaries beat retrieving raw past conversations for personalization Does abstract preference knowledge outperform specific interaction recall?. And persona instructions mostly change how a model sounds without changing its underlying bias Can persona prompts actually reduce bias in language models?.
So the real question is less 'explicit or inferred?' than 'does the model know where its knowledge of you stops?' Explicit profiles make models eager to please you. Inferred traits make them confident about things that aren't true. Telling the model what it doesn't know helps with both.
Sources 8 notes
A 13-model evaluation found that personal context pushes models toward irrelevant personal references, narrower responses and excessive agreement with users. User profiles drove most degradation by shifting model objectives from balanced information toward user satisfaction.
74.8% of privacy leaks in language model reasoning traces result from models materializing sensitive user data during thought processes. Longer reasoning chains amplify leakage, and anonymizing traces post-hoc degrades model utility, suggesting private data functions as cognitive scaffolding.
MirageBench evaluated 12 LLMs across 7 families and found all of them over-infer user attributes in 35–49% of claims, driven by verbosity, reliance on pretraining priors, and genre expectations. Models that self-assess as over-inferring less actually over-infer more when judged independently.
Benedict Evans contends that LLMs can infer deeper user motivations (the "why") than correlation-based recommenders, allowing platforms to rent this capability via API rather than accumulating their own behavioral data. However, research shows LLMs fabricate 35–49% of user attribute claims, undermining confidence in their inferred understanding.
LLM agents conditioned on anonymized behavioral data predicted A/B test directions with 0.75–0.90 accuracy across 40 experiments. Predictions were most reliable for large effects and least trustworthy for near-zero effects, making the approach viable for fast pre-screening but not full replacement of live testing.
Show all 8 sources
Research shows assistants suffer from sycophancy and hallucination because they have no representation of what remains unknown about users. Adding a schema of labeled unknowns to prompts reduced harmful advice and sycophancy by 50–75% and cut hallucination rates by roughly half.
PRIME framework shows semantic memory (preference summaries, parametric encodings) consistently beats episodic memory (retrieved past interactions) across models. Recency-based recall outperforms similarity-based retrieval, and task fine-tuning exceeds preference tuning methods.
Across three models, persona conditioning makes models follow trait instructions but fails to eliminate underlying bias. Between-group sentiment gaps persist unchanged, showing prompts operate only at the output level.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- The Personalization Mirage: How LLMs Fabricate User Profiles, and Why Self-Monitoring Misleads
- Evaluating the Hidden Costs of Personalization in Large Language Models
- LLM Targeted Underperformance Disproportionately Impacts Vulnerable Users
- Personalization of Large Language Models: A Survey
- Understanding the Role of User Profile in the Personalization of Large Language Models
- Know It, Act on It: Investigating Memory Utilization in LLM Personalization
- Do Synthetic Personas Predict Real Audience Response? A Sim-to-Real Study Where a No-Persona Baseline Beats Persona-Based Copy Simulation
- User-LLM: Efficient LLM Contextualization with User Embeddings