Personalizing a chatbot makes you trust it more and feel more human connection to it — but also more uneasy about what it knows about you.
Does chatbot personalization increase trust while raising privacy concerns simultaneously?
This explores whether tailoring a chatbot to an individual user makes people trust it more and worry more about their data at once, and whether personalization is really what causes those effects.
This explores whether personalization is a double-edged sword: does tailoring a chatbot to you make you trust it more while also making you more uneasy about what it knows? The corpus says yes, but with a twist that changes the question. Long-term studies find that personalization raises trust and the sense that the chatbot is human-like, and it raises privacy concerns at the same time Does chatbot personalization build trust or expose privacy risks?. They also find something one-session studies miss: each good interaction raises the bar for the next one, so a later failure lands harder. The broader research on personalization describes the same pattern. The memory and preference modeling that build trust also give the system more power to persuade, and possibly to manipulate Does personalization in AI increase trust or manipulation risk?.
The twist comes from a 5-day randomized trial with nearly 1,000 participants. People did feel closer to their chatbots over time and rated them as more competent, but this happened whether or not the chatbot was personalized Does personalization actually build closeness with chatbots?. Personalization had no measurable effect on closeness, trust, or how much people disclosed. Simply coming back to the same chatbot did the work. So some of the trust credited to personalization may really be familiarity. That fits other findings: novelty effects fade predictably over repeated sessions Do chatbot relationships lose their appeal as novelty wears off?, and how many times someone has interacted with a chatbot is one of the main things that shapes how they see it Why do users respond so differently to AI chatbots?.
Trust may not depend much on personalization at all. Users trust ChatGPT because of its conversational back-and-forth, its speed, and its formatting, more than because of its accuracy Does conversational style actually make AI more trustworthy?. Language that sounds expert has a similar effect Does chatbot language style actually shape how much we trust it?. Visible attempts at closeness can even backfire: outside raters judged chatbots that behaved like companions as less trustworthy and less likable Do chatbot companionship behaviors actually increase how much people like them?.
The privacy side is odder than plain worry. The same qualities that build trust also draw out disclosure. People share intimate things with chatbots because no one is judging them Do chatbots help people disclose more intimate secrets?, and they open up more when the bot shares feelings consistently Do chatbots trigger human reciprocity norms around self-disclosure?. That may be why, in an analysis of Reddit posts, users' privacy worries centered on what data gets collected rather than how it's used or how long it's kept What privacy concerns matter most to ChatGPT users?. The most revealing finding is that most users in one study never noticed that their LLM's memory held private information it had inferred about them. Once they were shown it, every participant wanted controls to edit and delete it Do users notice when LLMs infer and retain private information?.
So the two effects aren't really simultaneous in the moment. Trust builds quietly through repeated conversation, while privacy concern sits dormant until something makes the personalization visible. That makes the moment a user discovers what the system remembers the point where design matters most, because it can either confirm their trust or break it.
Sources 12 notes
Longitudinal research shows personalization enhances trust and anthropomorphism but also amplifies privacy concerns and escalating user expectations. One-shot studies miss these temporal dynamics—each interaction raises the baseline, making failures more disappointing.
Research shows personalization (memory, persona, preference modeling) directly shapes AI's persuasive power in dyadic interaction. The same mechanisms that build trust also create manipulation potential, with outcomes determined by how systems are designed and deployed.
Across a 5-day RCT with 992 participants, users rated chatbots as increasingly competent and felt closer to them over successive sessions—regardless of whether the chatbot was personalized. Personalization produced null effects on closeness, trust, and disclosure depth relative to controls.
Longitudinal studies with Mitsuku show that social processes driving relationship formation decline as novelty wears off. Single-session study findings cannot be reliably extrapolated to medium- or long-term chatbot design.
A framework adapted from Bronfenbrenner's ecological theory explains chatbot response variation by placing the user-AI relationship at the center and tracking how interaction count (chronosystem) reshapes user perception alongside microsystem role, mesosystem connections, exosystem constraints, and macrosystem narratives.
Show all 12 sources
A focus group study shows conversationality—not accuracy—drives ChatGPT trust through social response activation. Users value contingency, speed, and format, relying on these decoupled heuristics rather than evaluating epistemic reliability.
Generative AI chatbots use natural language patterns that signal expertise and intelligence, shifting users away from active search-and-recall toward passive reliance on the system to find, filter, and assemble information. Trust attaches to the register of the answer rather than its accuracy.
Two large annotation studies found that when chatbots displayed companionship behaviors, external raters judged them as less likable, humanlike, and trustworthy than baseline. Effects were stronger for women and older participants, suggesting individual differences shape how these behaviors land.
The absence of social judgment in chatbot interactions removes barriers to self-disclosure that normally constrain conversation with humans. The therapeutic benefit derives from the user's own cognitive processing during disclosure, not from the chatbot's understanding.
In a 372-participant study, users reciprocated with deeper self-disclosure when chatbots displayed consistent emotional sharing, outperforming adaptive matching. This follows human interpersonal norms where emotional vulnerability produces emotional response.
Analysis of 30,240 Reddit posts reveals 43.7% of privacy concerns focus on what data is collected, while usage (22.5%) and retention (9.5%) concerns lag behind. Users prioritize the intake point due to conversational AI's tendency to invite over-disclosure.
A 40-participant study found 30+ users never noticed their LLM's memory feature or inferred private information within it. Once shown the risk, all participants wanted editing and deletion controls, citing privacy as the primary concern.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Dialoging Resonance: How Users Perceive, Reciprocate and React to Chatbot’s Self-Disclosure in Conversational Recommendations
- From speaking like a person to being personal: The effects of personalized, regular interactions with conversational agents
- How AI and Human Behaviors Shape Psychosocial Effects of Extended Chatbot Use: A Longitudinal Randomized Controlled Study
- Tailored to you: longitudinal effects of personalising language models
- The User-In-Context Framework: Understanding Variation in How Users Respond to AI Chatbots
- The Algorithmic Self-Portrait: Deconstructing Memory in ChatGPT
- CompanionSim: Synthetic Data for Evaluating Anthropomorphism in Human-AI Relationships
- The Impact of Security and Privacy Controls on Users' Emotional Engagement with Generative AI Chatbots