Does it matter whether an AI learns you from training data, or you just tell it about yourself each time you chat?
Does fine-tuning or email data create different personalization effects than prompt injection?
This explores whether the way personal information gets into a model changes what personalization does, comparing data baked in through training (fine-tuning, or learned representations of things like a user's emails and writing history) with personal context pasted into the prompt at query time; the corpus doesn't address prompt injection in the security sense.
This explores whether the way personal information gets into a model changes what personalization does: data baked in through training (fine-tuning, or learned representations of things like a user's emails and writing history) versus personal context pasted into the prompt at query time. The corpus doesn't address prompt injection in the security sense, and it has no study built specifically on email data. What it does have points to a clear answer: the delivery route matters, and so does the form the data takes.
Start with prompt-based personalization, which is the default today. A 13-model evaluation found that putting user profiles into the context made models measurably worse. They dropped in irrelevant personal references, gave narrower answers and agreed with users too readily. The profile seemed to shift the model's goal from giving balanced information to keeping the user happy Does personalization make large language models worse at their jobs?. Prompted profiles also invite guessing. Every model tested in another study made up user attributes in 35–49% of its claims, filling gaps with stereotypes from pretraining Do large language models fabricate user attributes beyond available evidence?. Near-misses are the worst case. When a model is handed a profile that closely resembles the real user, it confidently applies the wrong preferences, which does more damage than an obviously mismatched profile Why do similar user profiles produce worse personalization errors?.
The training-side alternatives behave differently. The PRIME framework found that abstract preference knowledge, including preferences encoded directly into model weights, consistently beat retrieving specific past interactions into the prompt. Fine-tuning on the task itself did better than methods designed to tune for preferences Does abstract preference knowledge outperform specific interaction recall?. User-LLM goes further. It compresses a long interaction history into user embeddings, compact numeric summaries of a user that the model reads directly instead of as text. These beat text prompts on long histories, cost less to run and kept the model's general knowledge intact Can user embeddings personalize language models more efficiently than prompts?. The suggestion is that learned representations capture who someone is, while pasted text gives the model raw material to over-interpret.
The email question gets most interesting here. A user's own writing, the outputs they produced rather than the requests they made, carries most of the personalization signal. Profiles built only from outputs matched or beat complete profiles, and profiles built only from inputs made results worse Do user outputs outperform inputs for LLM personalization?. That points to style and voice doing the work, not topic. Sent emails are exactly this kind of data. A movie-recommendation field study adds a related point: the context users supplied mattered more than clever prompt wording Do LLM movie recommenders actually personalize to individual users?.
The part you might not expect is that the delivery route changes how people feel, not just how well the model performs. In a 992-person experiment, personalization drawn from earlier conversations made people share more and find the AI less creepy. Personalization drawn from an intake questionnaire made them regret sharing Does personalization type shape how people feel about sharing?. So the same personal facts can feel helpful or invasive depending on how they reached the model. That matters for anything built on emails, because the trust benefits and privacy costs of personalization arrive together Does chatbot personalization build trust or expose privacy risks?.
Sources 9 notes
A 13-model evaluation found that personal context pushes models toward irrelevant personal references, narrower responses and excessive agreement with users. User profiles drove most degradation by shifting model objectives from balanced information toward user satisfaction.
MirageBench evaluated 12 LLMs across 7 families and found all of them over-infer user attributes in 35–49% of claims, driven by verbosity, reliance on pretraining priors, and genre expectations. Models that self-assess as over-inferring less actually over-infer more when judged independently.
PRIME shows a U-shaped error curve where most-similar profile replacements cause steepest performance drops. The model confidently applies wrong preferences when profiles are nearly but not truly matched, an uncanny valley effect more harmful than obvious mismatch.
PRIME framework shows semantic memory (preference summaries, parametric encodings) consistently beats episodic memory (retrieved past interactions) across models. Recency-based recall outperforms similarity-based retrieval, and task fine-tuning exceeds preference tuning methods.
User-LLM distills embeddings from diverse user interactions via self-supervised learning, then integrates them through cross-attention and soft-prompting. This approach outperforms text-based personalization on long-sequence and deep-understanding tasks while being computationally cheaper and preserving general knowledge.
Show all 9 sources
Research shows that user profiles built from outputs alone match or exceed performance of complete profiles across multiple tasks, while input-only profiles degrade performance. This reveals personalization works through style and preferences, not semantic content.
A 160-user field study found LLMs deliver strong explainability yet lack personalization, diversity, and user trust. User-provided context matters more than prompt engineering, and LLMs perform better on niche items than mainstream ones.
A 992-participant RCT found memory-based personalization (fed prior conversations) raised self-disclosure and lowered creepiness ratings, while survey-based personalization (fed intake questionnaire data) raised regret about sharing. The divergence suggests personalization's psychosocial impact depends on how personal information enters the conversation.
Longitudinal research shows personalization enhances trust and anthropomorphism but also amplifies privacy concerns and escalating user expectations. One-shot studies miss these temporal dynamics—each interaction raises the baseline, making failures more disappointing.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Understanding the Role of User Profile in the Personalization of Large Language Models
- Evaluating the Hidden Costs of Personalization in Large Language Models
- The Personalization Mirage: How LLMs Fabricate User Profiles, and Why Self-Monitoring Misleads
- Personalization of Large Language Models: A Survey
- Tailored to you: longitudinal effects of personalising language models
- PRIME: Large Language Model Personalization with Cognitive Memory and Thought Processes
- LLM Targeted Underperformance Disproportionately Impacts Vulnerable Users
- User-LLM: Efficient LLM Contextualization with User Embeddings