Before anyone shows you what an AI chatbot remembers about you, what do you assume it's keeping?
What privacy expectations do users hold before learning what systems store?
This explores what people assume an AI chatbot remembers and protects about them before anyone shows them what it actually stores, and how those assumptions change once they find out.
This explores what people assume an AI chatbot remembers about them before anyone shows them what it actually keeps. The surprising answer from the corpus is that most users don't hold clear privacy expectations at all. In one 40-person study, more than 30 participants had never noticed that their LLM had a memory feature, or that it had inferred private facts about them and saved them Do users notice when LLMs infer and retain private information?. Where people do have a picture, they disagree. Interviews with ChatGPT users found four incompatible mental models. Some think memory lasts only for the session. Some think it is part of training data. Some think it is active processing. The rest simply don't know. Which model a person holds shapes how careful they are What mental models do users hold about LLM memory?.
That vagueness helps explain where worry goes. In a large analysis of Reddit posts, users' concerns clustered at the point of intake: 43.7% were about what data gets collected, while only 9.5% were about how long it is kept What privacy concerns matter most to ChatGPT users?. People guard the moment of typing and give little thought to what happens next, which is exactly where memory and inference operate. The same discussions show people settling into distinct attitudes before they have full information. Some actively adjust settings, some go looking for answers, some trade privacy for convenience, and some give up and accept data sharing as inevitable What different ways do people handle AI privacy concerns?.
A quieter finding is that feeling safe and being safe come apart. Participants who edited chatbot transcripts by hand felt just as protected as those who used AI-suggested redactions, yet they left far more identifying information behind Why do people feel safe sharing when they haven't actually removed sensitive information?. In a 300-person survey, people chose AI-drafted replies that leaked private information over their own drafts 55% of the time, apparently without noticing the breach Do people notice when AI agents leak private information?. How information enters the system also shapes comfort more than how much is stored. When a chatbot drew on past conversations, people disclosed more and found it less creepy. When it drew on an intake questionnaire, they regretted sharing more Does personalization type shape how people feel about sharing?. Over time, personalization builds trust and privacy unease at the same pace Does chatbot personalization build trust or expose privacy risks?.
The gap matters because systems hold more than users picture. Models make up 35–49% of the claims they produce about user attributes, going past the available evidence Do large language models fabricate user attributes beyond available evidence?. Reasoning models restate private details inside their own thought process, which becomes a fresh source of leaks Do reasoning traces actually expose private user data?. What a system stores, then, includes guesses about you, not only what you typed.
What happens after people learn is fairly consistent. Once shown what was stored, every participant in one study wanted editing and deletion controls Do users notice when LLMs infer and retain private information?. Others asked for consent when a memory is created, a way to review stored memories, and limits on how memories are used What memory controls do users actually want from LLMs?. One proposed design splits the work so users state their preferences and the AI carries out the edits, to keep that control from becoming a chore Does splitting privacy work between users and AI reduce cognitive burden?. The takeaway: people's privacy expectations mostly don't exist until someone shows them what was stored, and good design may depend more on that first reveal than on default settings.
Sources 12 notes
A 40-participant study found 30+ users never noticed their LLM's memory feature or inferred private information within it. Once shown the risk, all participants wanted editing and deletion controls, citing privacy as the primary concern.
Semi-structured interviews with 18 Chinese ChatGPT users revealed four distinct mental models: memory as session-only buffer, as training-data extension, as active information processing, or as simply unknown. These models directly influence privacy awareness and user behavior.
Analysis of 30,240 Reddit posts reveals 43.7% of privacy concerns focus on what data is collected, while usage (22.5%) and retention (9.5%) concerns lag behind. Users prioritize the intake point due to conversational AI's tendency to invite over-disclosure.
Qualitative analysis of Reddit discussions identified four recurring attitudes: proactive users actively adjust settings, inquisitive users seek information about data practices, convenience-prioritizers downplay risks for benefits, and resigned users accept data sharing as inevitable. The typology emerged from coding user behavior descriptions rather than survey responses.
Across 188 participants, those manually editing chatbot transcripts reported equivalent privacy confidence as those using AI suggestions, yet left substantially more identifiable information in their final transcripts. This decoupling suggests comfort reflects editing interface design rather than actual data exposure.
Show all 12 sources
In a 300-person survey, participants preferred AI-generated replies containing privacy violations 55% of the time, up from 15.7% baseline. Researchers attribute this to overtrust and failure to notice contextual privacy breaches rather than deliberate trade-offs.
A 992-participant RCT found memory-based personalization (fed prior conversations) raised self-disclosure and lowered creepiness ratings, while survey-based personalization (fed intake questionnaire data) raised regret about sharing. The divergence suggests personalization's psychosocial impact depends on how personal information enters the conversation.
Longitudinal research shows personalization enhances trust and anthropomorphism but also amplifies privacy concerns and escalating user expectations. One-shot studies miss these temporal dynamics—each interaction raises the baseline, making failures more disappointing.
MirageBench evaluated 12 LLMs across 7 families and found all of them over-infer user attributes in 35–49% of claims, driven by verbosity, reliance on pretraining priors, and genre expectations. Models that self-assess as over-inferring less actually over-infer more when judged independently.
74.8% of privacy leaks in language model reasoning traces result from models materializing sensitive user data during thought processes. Longer reasoning chains amplify leakage, and anonymizing traces post-hoc degrades model utility, suggesting private data functions as cognitive scaffolding.
An 18-person interview study found users want explicit consent at memory generation, management interfaces to review and edit memories, purpose limits on usage, and agency over inferred data. Participants actively perform privacy calculus rather than passively accept memory's trade-offs.
MemoAnalyzer's design divides memory-privacy labor: users state preferences, AI executes edits. This role split is proposed as a way to reduce cognitive load and improve perceived agency, though validation metrics and conflict-resolution processes remain undescribed.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- The Algorithmic Self-Portrait: Deconstructing Memory in ChatGPT
- Understanding Users' Privacy Perceptions Towards LLM's RAG-based Memory
- "Ghost of the past": identifying and resolving privacy leakage from LLM's memory through proactive user interaction
- Privacy Leakage Overshadowed by Views of AI: A Study on Human Oversight of Privacy in Language Model Agent
- Disclose with Care: Designing Privacy Controls in Interview Chatbots
- Autonomy Reshapes How Personalization Affects Privacy Concerns and Trust in LLM Agents
- Understanding Users' Security and Privacy Concerns and Attitudes Towards Conversational AI Platforms
- Relational Gains, Privacy Strains: Exploring Users' Perceptions and Experiences with ChatGPT's Memory Feature