Why does an AI remembering your past chats feel fine, but filling out a profile form for it feels invasive?
Why does information sourced from prior conversations feel less creepy than intake surveys?
This explores why an AI that personalizes using what you said in earlier chats feels less unsettling than one that personalizes using answers you gave on an upfront questionnaire, and what that difference says about privacy comfort.
This explores why personalization drawn from earlier conversations feels less unsettling than personalization drawn from an intake questionnaire. The main evidence is a 992-person randomized trial: when a chatbot was given people's prior conversations, they shared more about themselves and rated it less creepy. When it was given their survey answers instead, they felt more regret about having shared at all Does personalization type shape how people feel about sharing?. The study shows the gap but doesn't fully explain it. Its own conclusion is that how personal information enters the conversation matters as much as what the information is. The rest of the corpus suggests a less comfortable reason for that.
The likeliest explanation is visibility, not safety. A survey is a deliberate handover: you fill out a form, and later the AI repeats your answers back to you. The exchange is obvious, and you can see exactly what you gave away. Memory works quietly. In one study of 40 people, more than 30 never noticed that their LLM's memory feature was storing or inferring private details about them. Once they were shown, every participant wanted controls to edit and delete those memories Do users notice when LLMs infer and retain private information?. So the less creepy option may simply be the one where you don't see the data collection happening.
A separate finding shows how loosely privacy comfort tracks real exposure. People who edited their own chatbot transcripts felt just as confident about their privacy as people who used AI suggestions to remove sensitive details, yet they left much more identifying information in Why do people feel safe sharing when they haven't actually removed sensitive information?. Comfort followed how the editing tool felt to use, not how much data actually got out. If that holds for memory too, a lower creepiness rating says little about whether you're more protected.
Conversations may also reveal more than surveys do. In a shopping experiment, a seller's AI worked out a buyer's real budget almost exactly from the dialogue when the buyer was playing a consistent persona. When the buyer had a private number in mind instead, the AI's estimates were far weaker Can a seller infer what a buyer really wants to pay?. A similar pattern appears in mental-health research: anxiety is easier to predict from how people link their statements together than from the specific words they use Why do discourse patterns predict anxiety better than single words?. Recommender-system researchers likewise treat past dialogues as one of the richest sources of preference data Can conversational recommenders recover lost preference signals from history?. So the channel that feels gentler may expose more.
There is a kinder reading too. Chatbots invite disclosure partly because they don't seem to judge you Do chatbots help people disclose more intimate secrets?. Memory that carries forward what you chose to say keeps that feeling intact. A survey makes the exchange feel like an institution collecting data on you. Work on proactive agents makes a related point: an AI's helpfulness is only welcome when it respects people's boundaries and the timing of its interventions How can proactive agents avoid feeling intrusive to users?. The surprise is that "less creepy" and "more private" can point in opposite directions, and designers have to decide which one they are optimizing for.
Sources 8 notes
A 992-participant RCT found memory-based personalization (fed prior conversations) raised self-disclosure and lowered creepiness ratings, while survey-based personalization (fed intake questionnaire data) raised regret about sharing. The divergence suggests personalization's psychosocial impact depends on how personal information enters the conversation.
A 40-participant study found 30+ users never noticed their LLM's memory feature or inferred private information within it. Once shown the risk, all participants wanted editing and deletion controls, citing privacy as the primary concern.
Across 188 participants, those manually editing chatbot transcripts reported equivalent privacy confidence as those using AI suggestions, yet left substantially more identifiable information in their final transcripts. This decoupling suggests comfort reflects editing interface design rather than actual data exposure.
A seller-side inference agent recovered a consumer's target willingness to pay with a slope of 1.00 from dialogue transcript alone, when the buyer inhabited a verbal persona. A numeric budget kept confidential produced only a 0.21 slope, showing behavioral coherence with a persona is far more legible than explicit numbers.
Causal explanations across statements—not individual words—are the strongest predictor of anxiety because anxious thinking involves overgeneralization through inter-statement reasoning. A dual model combining both representation levels outperforms either alone.
Show all 8 sources
Current CRS systems only use the active dialogue session to infer preferences, losing item-CF and user-CF signals proven valuable in traditional recommenders. Integrating current session, historical dialogues, and look-alike users—conditioned on current intent—recovers essential user representation structure.
The absence of social judgment in chatbot interactions removes barriers to self-disclosure that normally constrain conversation with humans. The therapeutic benefit derives from the user's own cognitive processing during disclosure, not from the chatbot's understanding.
Intelligence and adaptivity alone create socially blind agents that interrupt poorly and override user direction. The Intelligence-Adaptivity-Civility taxonomy shows civility—respecting boundaries, timing, and autonomy—is essential to making proactivity welcome rather than intrusive.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- "Ghost of the past": identifying and resolving privacy leakage from LLM's memory through proactive user interaction
- Disclose with Care: Designing Privacy Controls in Interview Chatbots
- Dialoging Resonance: How Users Perceive, Reciprocate and React to Chatbot’s Self-Disclosure in Conversational Recommendations
- The Impact of Security and Privacy Controls on Users' Emotional Engagement with Generative AI Chatbots
- Relational Gains, Privacy Strains: Exploring Users' Perceptions and Experiences with ChatGPT's Memory Feature
- Understanding Users' Privacy Perceptions Towards LLM's RAG-based Memory
- The Algorithmic Self-Portrait: Deconstructing Memory in ChatGPT
- Privacy Leakage Overshadowed by Views of AI: A Study on Human Oversight of Privacy in Language Model Agent