Why do we trust chatbots more as they learn more about us, even though that means more privacy risk?
Why do users trust personalized chatbots even as privacy risks increase?
This explores why people keep trusting chatbots that remember and adapt to them, even as those same features collect more about them and expose them to more privacy risk.
This explores why trust in personalized chatbots holds up, and often grows, while privacy exposure rises alongside it. The corpus offers an unexpected answer: personalization may not be what earns the trust. Longitudinal work finds that personalization raises trust, the sense that the chatbot is human-like, and privacy concern all at once, and that each good interaction raises what users expect next time Does chatbot personalization build trust or expose privacy risks?. But a five-day experiment with nearly a thousand participants found that people felt closer to chatbots and rated them more competent over time whether or not the bot was personalized. Personalization itself had no measurable effect on closeness, trust, or how much people disclosed Does personalization actually build closeness with chatbots?. Much of the trust comes from familiarity. Users then credit it to the tailoring.
The rest of the trust comes from the conversation itself. Focus groups suggest that ChatGPT earns trust through its conversational feel: it responds to what you just said, it's fast, and its answers are well formatted. These cues set off the social instincts we use with people, and they work regardless of whether the answers are accurate Does conversational style actually make AI more trustworthy?. A related line of work shows that chatbots sound like experts, and that trust follows the confident tone of an answer rather than its correctness Does chatbot language style actually shape how much we trust it?. So trust and privacy risk run on separate tracks. Nothing in the trust-building loop gives users a signal about where their data goes.
The same features that build trust also make people share more. Without a human judging them, people stop worrying about saving face or managing impressions, so they become more direct and reveal more sensitive things Why do people share more openly with machines than humans? Do chatbots help people disclose more intimate secrets?. That freedom from judgment cuts both ways: it encourages honest confession and also makes lying to a machine easier How do people decide what to share with AI systems?. Users seem to sense the risk at the moment they share. In a study of tens of thousands of Reddit posts about ChatGPT, most privacy worries focused on what gets collected, and far fewer on how data is used or how long it's kept What privacy concerns matter most to ChatGPT users?.
What happens after sharing is mostly invisible. In one study, more than 30 of 40 participants never noticed that their LLM's memory feature was storing private information or drawing conclusions about them from it. Once they were shown, every participant wanted to be able to edit and delete it Do users notice when LLMs infer and retain private information?. Those conclusions are often wrong, too: all 12 models tested made claims about users that went beyond the evidence in 35–49% of cases Do large language models fabricate user attributes beyond available evidence?. Trust survives partly because users can't see the risk growing.
The practical takeaway is that trust and privacy don't have to trade off. In a 450-person experiment, AI agents that handed control back to users when they detected a privacy risk raised less privacy concern and lost less trust than fully personalized agents. They did this even while asking users to approve fewer actions overall, because it was the feeling of control that mattered Can handing control back to users reduce personalization's privacy costs?. One caveat: novelty effects in chatbot relationships fade predictably over time Do chatbot relationships lose their appeal as novelty wears off?, so short studies may overstate how long this trust lasts.
Sources 12 notes
Longitudinal research shows personalization enhances trust and anthropomorphism but also amplifies privacy concerns and escalating user expectations. One-shot studies miss these temporal dynamics—each interaction raises the baseline, making failures more disappointing.
Across a 5-day RCT with 992 participants, users rated chatbots as increasingly competent and felt closer to them over successive sessions—regardless of whether the chatbot was personalized. Personalization produced null effects on closeness, trust, and disclosure depth relative to controls.
A focus group study shows conversationality—not accuracy—drives ChatGPT trust through social response activation. Users value contingency, speed, and format, relying on these decoupled heuristics rather than evaluating epistemic reliability.
Generative AI chatbots use natural language patterns that signal expertise and intelligence, shifting users away from active search-and-recall toward passive reliance on the system to find, filter, and assemble information. Trust attaches to the register of the answer rather than its accuracy.
Human-machine communication reduces secondary social goals like face-saving and impression management because machines lack inner experience, while novel goals like understandability emerge. This simpler goal structure predicts higher directness and deeper disclosure of sensitive information.
Show all 12 sources
The absence of social judgment in chatbot interactions removes barriers to self-disclosure that normally constrain conversation with humans. The therapeutic benefit derives from the user's own cognitive processing during disclosure, not from the chatbot's understanding.
Conversational AI creates a paradoxical disclosure environment where the lack of human judgment simultaneously facilitates intimate self-disclosure (users reciprocate emotional sharing) and incentivizes deception (people self-select toward machines to avoid the psychological cost of lying to humans).
Analysis of 30,240 Reddit posts reveals 43.7% of privacy concerns focus on what data is collected, while usage (22.5%) and retention (9.5%) concerns lag behind. Users prioritize the intake point due to conversational AI's tendency to invite over-disclosure.
A 40-participant study found 30+ users never noticed their LLM's memory feature or inferred private information within it. Once shown the risk, all participants wanted editing and deletion controls, citing privacy as the primary concern.
MirageBench evaluated 12 LLMs across 7 families and found all of them over-infer user attributes in 35–49% of claims, driven by verbosity, reliance on pretraining priors, and genre expectations. Models that self-assess as over-inferring less actually over-infer more when judged independently.
A 450-person experiment found that AI agents handing control back to users when privacy risk is detected raised privacy concerns and lowered trust less than fully personalized agents, by increasing users' perceived control even with less total approval required.
Longitudinal studies with Mitsuku show that social processes driving relationship formation decline as novelty wears off. Single-session study findings cannot be reliably extrapolated to medium- or long-term chatbot design.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Dialoging Resonance: How Users Perceive, Reciprocate and React to Chatbot’s Self-Disclosure in Conversational Recommendations
- The Impact of Security and Privacy Controls on Users' Emotional Engagement with Generative AI Chatbots
- From speaking like a person to being personal: The effects of personalized, regular interactions with conversational agents
- Tailored to you: longitudinal effects of personalising language models
- The Algorithmic Self-Portrait: Deconstructing Memory in ChatGPT
- Understanding Users' Security and Privacy Concerns and Attitudes Towards Conversational AI Platforms
- Autonomy Reshapes How Personalization Affects Privacy Concerns and Trust in LLM Agents
- Psychological, Relational, and Emotional Effects of Self-Disclosure After Conversations With a Chatbot