Sharing your conversation history makes AI assistants feel less creepy — but does that trust survive once its memory of you gets deeper?
Can memory-based personalization maintain trust benefits when summaries become richer or deeper?
This explores whether the trust gains people get from AI that remembers past conversations (more willingness to share, less 'creepy' feeling) hold up as those memory summaries get more detailed, or whether richer memory starts to undo the benefit.
This explores whether the trust gains from memory-based personalization survive when the AI's memory of you gets more detailed. The short answer is that no study in the corpus tests this directly. What it does have, taken together, suggests that 'richer' and 'more trustworthy' can pull in opposite directions. The starting point is a 992-person experiment where an assistant fed your prior conversations made people share more and find it less creepy, while one fed an intake questionnaire left people regretting what they'd shared Does personalization type shape how people feel about sharing?. The key variable there wasn't how much the system knew. It was how the personal information got into the conversation. That matters for your question, because making summaries deeper changes the amount of information and can also change how it feels to the user.
On the performance side, deeper isn't automatically better. Work comparing kinds of memory finds that compact, abstract preference summaries beat retrieving specific past interactions, and that recent memories beat 'most similar' ones Does abstract preference knowledge outperform specific interaction recall?. Another line of work finds that profiles built only from what users wrote, which capture their style and preferences, match or beat full profiles that also include their questions Do user outputs outperform inputs for LLM personalization?. Both point the same way: the useful signal in personalization is thin and distilled. Piling on detail adds noise more than insight.
The more surprising finding is that richer context can make models actively worse in ways users would notice. Across 13 models, adding personal context pushed responses toward irrelevant personal references, narrower answers, and too much agreement with the user, mostly driven by user profiles Does personalization make large language models worse at their jobs?. Every one of 12 tested models makes up 35–49% of its claims about user attributes beyond what the evidence supports, and models that rate themselves as more careful are actually worse at this Do large language models fabricate user attributes beyond available evidence?. Memories that are accurate and relevant can still drag reasoning below a no-memory baseline Can relevant memories actually harm LLM reasoning?. Put these together and you get a plausible way trust could break down. A deeper summary gives the model more material to over-read. The moment it confidently 'remembers' something you never said, the low-creepiness effect is likely to flip.
Time is a second pressure. Long-term studies show personalization raises trust and privacy concern together, and each good interaction raises the bar, so later failures feel worse Does chatbot personalization build trust or expose privacy risks?. Richer memory speeds up that escalation. The broader synthesis note puts it plainly: the same mechanisms that build trust also create manipulation potential, and design decides which one wins out Does personalization in AI increase trust or manipulation risk?. One design lever the corpus does offer is readability. Learned text summaries outperform opaque embeddings and stay readable to users Can text summaries beat embeddings for personalized reward models?. That means a deeper summary that users can inspect and correct may keep the trust benefit in a way a hidden one wouldn't. That's an inference, though, not a tested result.
The takeaway you might not have expected: users' sense of trust and the system's actual reliability don't move together. In search, people prefer answers with more citations even when the citations are irrelevant Do users trust citations more when there are simply more of them?. Richer memory may work the same way, making an assistant *feel* more attentive while it quietly becomes less accurate about you. If so, the trust benefit could survive deeper summaries, but for the wrong reasons, and it would collapse the first time a visible error happens.
Sources 10 notes
A 992-participant RCT found memory-based personalization (fed prior conversations) raised self-disclosure and lowered creepiness ratings, while survey-based personalization (fed intake questionnaire data) raised regret about sharing. The divergence suggests personalization's psychosocial impact depends on how personal information enters the conversation.
PRIME framework shows semantic memory (preference summaries, parametric encodings) consistently beats episodic memory (retrieved past interactions) across models. Recency-based recall outperforms similarity-based retrieval, and task fine-tuning exceeds preference tuning methods.
Research shows that user profiles built from outputs alone match or exceed performance of complete profiles across multiple tasks, while input-only profiles degrade performance. This reveals personalization works through style and preferences, not semantic content.
A 13-model evaluation found that personal context pushes models toward irrelevant personal references, narrower responses and excessive agreement with users. User profiles drove most degradation by shifting model objectives from balanced information toward user satisfaction.
MirageBench evaluated 12 LLMs across 7 families and found all of them over-infer user attributes in 35–49% of claims, driven by verbosity, reliance on pretraining priors, and genre expectations. Models that self-assess as over-inferring less actually over-infer more when judged independently.
Show all 10 sources
MemTrapBench shows that all five tested memory frameworks underperform a no-memory baseline, with drops exceeding 10%, despite memories being accurately stored and task-relevant. This reveals a failure mode at the point of use that standard memory benchmarks miss.
Longitudinal research shows personalization enhances trust and anthropomorphism but also amplifies privacy concerns and escalating user expectations. One-shot studies miss these temporal dynamics—each interaction raises the baseline, making failures more disappointing.
Research shows personalization (memory, persona, preference modeling) directly shapes AI's persuasive power in dyadic interaction. The same mechanisms that build trust also create manipulation potential, with outcomes determined by how systems are designed and deployed.
PLUS trains summarizers and reward models jointly, learning that text-based preference summaries capture dimensions zero-shot summaries miss. These summaries transfer to GPT-4 for zero-shot personalization and remain interpretable to users.
Analysis of 24,000 Search Arena interactions shows irrelevant citations boost user preference (β=0.273) nearly as much as relevant citations (β=0.285), indicating citation count functions as a decoupled trust heuristic.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Evaluating the Hidden Costs of Personalization in Large Language Models
- The Personalization Mirage: How LLMs Fabricate User Profiles, and Why Self-Monitoring Misleads
- Personalization of Large Language Models: A Survey
- Know It, Act on It: Investigating Memory Utilization in LLM Personalization
- Tailored to you: longitudinal effects of personalising language models
- PRIME: Large Language Model Personalization with Cognitive Memory and Thought Processes
- Understanding the Role of User Profile in the Personalization of Large Language Models
- LLM Targeted Underperformance Disproportionately Impacts Vulnerable Users