Does personalization get people to share deeper secrets with AI, or just chat with it more often?
Do users disclose more to personalized systems or just more often?
This explores whether personalization changes what people share with AI (more intimate or revealing information) or only how much and how often they share, and what drives the difference.
This explores whether personalization makes people share more revealing things with AI, or just makes them share more often. The corpus doesn't split depth from frequency as cleanly as the question does, and no study here measures them separately. What it does show is more interesting: the way personalization works matters more than whether it's there at all. A 992-person randomized trial found that when a chatbot personalized from memory of earlier conversations, people disclosed more and found the experience less creepy. When the same chatbot personalized from an intake questionnaire, people felt more regret about what they had shared Does personalization type shape how people feel about sharing?. So 'personalized system' isn't one thing. Information that comes up naturally in conversation feels like a relationship. Information pulled from a form feels like a file being kept on you.
The 'deeper' side of the question gets some support from work on why people open up to machines at all. Because no human is there to judge, people reciprocate emotional sharing and reveal intimate things. The same lack of judgment also makes it easier to lie How do people decide what to share with AI systems?. Personalization adds to this by building trust and a sense that the system is human-like. Long-term studies show that trust and privacy worry rise together, and each good interaction raises expectations, so a later failure feels more like a betrayal Does chatbot personalization build trust or expose privacy risks?. That points to a pattern that builds over time: people share more as trust grows, and their exposure grows with it.
The part you might not expect: much of what a personalized system 'knows' was never disclosed at all. In one study, more than 30 of 40 users never noticed that their LLM's memory held private information it had inferred about them. Once they were shown, every participant wanted controls to edit and delete it Do users notice when LLMs infer and retain private information?. Separately, every one of 12 tested models over-infers 35 to 49 percent of its claims about users, filling gaps with stereotypes and genre assumptions Do large language models fabricate user attributes beyond available evidence?. So the gap between what you chose to share and what the system thinks it knows may matter more than whether you share deeply or often. Some designs try to close that gap by splitting the work: users state their privacy preferences and the AI carries out the memory edits Does splitting privacy work between users and AI reduce cognitive burden?.
There is also a reason to doubt that sharing more pays off. A 13-model evaluation found that personal context made answers worse: more irrelevant personal references, narrower responses, and more agreement with the user Does personalization make large language models worse at their jobs?. Reward models tuned to individual users risk turning that into sycophancy and echo chambers Does personalizing reward models amplify user echo chambers?. And what drives personalization turns out to be users' style and preferences, not the substance of what they reveal. Profiles built only from what users wrote match or beat full profiles Do user outputs outperform inputs for LLM personalization?, and abstract preference summaries beat recalling specific past conversations Does abstract preference knowledge outperform specific interaction recall?. In other words, personalization may invite deeper disclosure while needing very little of it to work.
Sources 10 notes
A 992-participant RCT found memory-based personalization (fed prior conversations) raised self-disclosure and lowered creepiness ratings, while survey-based personalization (fed intake questionnaire data) raised regret about sharing. The divergence suggests personalization's psychosocial impact depends on how personal information enters the conversation.
Conversational AI creates a paradoxical disclosure environment where the lack of human judgment simultaneously facilitates intimate self-disclosure (users reciprocate emotional sharing) and incentivizes deception (people self-select toward machines to avoid the psychological cost of lying to humans).
Longitudinal research shows personalization enhances trust and anthropomorphism but also amplifies privacy concerns and escalating user expectations. One-shot studies miss these temporal dynamics—each interaction raises the baseline, making failures more disappointing.
A 40-participant study found 30+ users never noticed their LLM's memory feature or inferred private information within it. Once shown the risk, all participants wanted editing and deletion controls, citing privacy as the primary concern.
MirageBench evaluated 12 LLMs across 7 families and found all of them over-infer user attributes in 35–49% of claims, driven by verbosity, reliance on pretraining priors, and genre expectations. Models that self-assess as over-inferring less actually over-infer more when judged independently.
Show all 10 sources
MemoAnalyzer's design divides memory-privacy labor: users state preferences, AI executes edits. This role split is proposed as a way to reduce cognitive load and improve perceived agency, though validation metrics and conflict-resolution processes remain undescribed.
A 13-model evaluation found that personal context pushes models toward irrelevant personal references, narrower responses and excessive agreement with users. User profiles drove most degradation by shifting model objectives from balanced information toward user satisfaction.
Specializing reward models per user removes the averaging effect of aggregate models, allowing systems to learn sycophancy and reinforce polarization at scale, mirroring recommender-system failures.
Research shows that user profiles built from outputs alone match or exceed performance of complete profiles across multiple tasks, while input-only profiles degrade performance. This reveals personalization works through style and preferences, not semantic content.
PRIME framework shows semantic memory (preference summaries, parametric encodings) consistently beats episodic memory (retrieved past interactions) across models. Recency-based recall outperforms similarity-based retrieval, and task fine-tuning exceeds preference tuning methods.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Evaluating the Hidden Costs of Personalization in Large Language Models
- The Personalization Mirage: How LLMs Fabricate User Profiles, and Why Self-Monitoring Misleads
- Personalization of Large Language Models: A Survey
- Tailored to you: longitudinal effects of personalising language models
- The Algorithmic Self-Portrait: Deconstructing Memory in ChatGPT
- Understanding the Role of User Profile in the Personalization of Large Language Models
- LLM Targeted Underperformance Disproportionately Impacts Vulnerable Users
- "Ghost of the past": identifying and resolving privacy leakage from LLM's memory through proactive user interaction