Do users notice when LLMs infer and retain private information?
Explores whether people are aware that conversational AI systems can infer and store sensitive personal details from their interactions, and what happens when they learn this occurs.
"Ghost of the past" runs a semi-structured interview and five-day user study of people interacting with an LLM-based conversational agent's long-term memory feature, and reports three findings (F1–F3) about what those participants did and didn't notice. F1, "Memory Mechanism is Opaque": "Without explicit notification, 30/40 of users were unaware of the notification when memory and context were added," the management interface showed memories "in a plain manner, without highlighting the potential private information," and the panel itself was hard to find — "35/40 participants never clicked it." F2, "Users Lack the Privacy Awareness towards the Memory Mechanism": 36/40 never noticed private information in their memories before the study, and every participant echoed "I have never thought about these information usage and leakage patterns before." F3: once shown the risk, all participants wanted editing, adding, and deleting control, with privacy the top-cited motivator (35/40) ahead of accuracy (14/40).
The paper's own mechanism for why this stayed invisible: nothing distinguished direct disclosure (private information stated in one input or memory) from the riskier case the system was actually doing — information "inferred from multiple past inputs or memories" combined. Participants could follow that inference once it was explained to them, but none caught it unprompted, which the authors read as evidence of "in-transparency" rather than user carelessness. They also separate this risk from training-time memorization leakage: because conversational memory is retained longer, "the threats are more severe" than a single input's exposure, even though the underlying privacy mechanics are similar.
This sits upstream of Does chatbot personalization build trust or expose privacy risks?: that note describes privacy concern ratcheting up as personalization deepens, but this study shows the ratchet doesn't even start until a user is made aware memory exists — most participants here hadn't reached the point where rising trust and rising privacy concern could trade off against each other. It also names a distinct leakage path from Does repeated sensitive data in fine-tuning cause memorization?, which targets training-time memorization; this paper's risk is inference at use-time, across a user's own retained conversational memories. And it inverts Do language models know what they don't know about users?: that note is about the model's blind spot regarding the user, while this one documents the user's blind spot regarding the model.
The excerpt doesn't quantify how memory is actually combined to produce inferences, nor does it establish that these findings generalize beyond one interface's notification and panel design — participants were recruited by snowball sampling on a campus, skewing toward higher education, which the authors argue would make the opacity problem look better, not worse, than in a general population. What the evidence does support: passive, opt-out-style memory notices are not reaching users who have no prior reason to suspect inference is happening, so any memory feature claiming "the user is in control" needs affirmative, inference-level disclosure before storage, not just a settings panel.
Inquiring lines that read this note 48
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
Do engagement-optimized chatbots create dependencies that undermine user autonomy?- Does expected post-interview editing change how much personal detail people disclose during chatbot interviews?
- How does discovering retained details shift user trust in chatbots?
- Does novelty wear off differently when chatbots remember your preferences?
- Why do people report equal privacy comfort despite removing vastly different amounts of personal information?
- Can editing interfaces create dark patterns where users add sensitive details back after abstractions?
- Can disclosure budgets protect users if they only account for factual data types?
- What privacy risks emerge when AI stores psychological rather than factual data?
- How stable are user privacy attitudes when major AI events occur?
- What privacy safeguards should apply to memories users never explicitly requested?
- What privacy expectations do users hold before learning what systems store?
- How do AI systems define 'private information' or 'high-stakes action' in training?
- What six distinct patterns emerge when people review AI agent responses for privacy?
- Why does giving users full choice fail to prevent privacy leaks?
- How do privacy concerns translate into catching actual information breaches?
- How do major AI ecosystem events shift user privacy sentiment over time?
- What share of stated privacy concerns reflect actual platform behavior versus misconceptions?
- How can systems visualize where privacy risks originate in stored memories?
- Does default inaction explain why free-editing participants left more personal information behind?
- Can real-time privacy judgment during editing reveal what participants actually notice as sensitive?
- How much does interface design shape privacy behavior versus underlying privacy concerns?
- How do false mental models about AI systems affect user privacy behavior?
- Do users disclose more to personalized systems or just more often?
- Why do people with high AI trust fail to notice privacy leaks?
- Does giving users more control over AI agents actually improve privacy protection?
- Does giving users memory control actually reduce their privacy concerns?
- What strategies do users already employ to protect sensitive information?
- Can systems bound what gets disclosed per query without user consent?
- Do general populations show the same privacy awareness gaps as college-educated users?
- Can identity verification schemes preserve anonymity for high-stakes disclosures?
- How does unnoticed personalization affect user trust and privacy expectations?
- Do personalized chatbots increase both trust and privacy concerns simultaneously?
- Do personalized chatbots amplify confusion about privacy controls?
- Does personalization in chatbots increase trust and privacy concerns simultaneously?
- How do emotional connections to AI systems affect privacy decision-making?
- Does personalization in chatbots increase both trust and privacy concerns simultaneously?
- Why does information sourced from prior conversations feel less creepy than intake surveys?
- Can memory-based personalization increase self-disclosure without raising trust?
- Does the trust benefit of personalization justify its privacy costs?
- How does memory personalization increase both trust and privacy anxiety simultaneously?
- How does conversational memory pose different privacy risks than training-time memorization?
- Why do users trust personalized chatbots even as privacy risks increase?
Related concepts in this collection 6
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
-
Does chatbot personalization build trust or expose privacy risks?
Explores whether personalization features that increase user trust and social connection simultaneously heighten privacy concerns and create rising behavioral expectations over time.
that ratchet assumes awareness this study shows most users don't yet have
-
Does repeated sensitive data in fine-tuning cause memorization?
When language models train on the same private or proprietary data multiple times, how much do they end up memorizing and leaking that information at inference time? Understanding this risk is critical for organizations fine-tuning on confidential datasets.
a distinct, training-time leakage path contrasted with this paper's use-time inference risk
-
Do language models know what they don't know about users?
Explores whether AI assistants fail because they lack explicit awareness of their knowledge gaps about the person they're helping, and whether marking unknowns in prompts could reduce errors.
mirrors the model's blind spot about the user with the user's blind spot about the model
-
Can a smaller user model subset match full model performance?
Does selecting just a few key fields from a larger user model preserve personalization quality while reducing token transmission? This matters for keeping user data local and limiting what leaves the device per query.
offers a measurable disclosure ceiling that could operationalize the control these users asked for
-
What memory controls do users actually want from LLMs?
Interview participants weigh memory's personalization benefits against privacy risks like data leakage and profiling. The research explores what specific intervention points across memory's lifecycle would let users manage that tension.
Extends A's demand for proactive control by detailing lifecycle-wide controls wanted: consent, editing, and deletion
-
Does ChatGPT's memory feature feel like a relationship or a machine?
Users experience ChatGPT's memory as having machine-like qualities yet fostering relationship-building. But what happens when they discover exactly what it remembers about them, and does that discovery match their expectations?
Extends A's discovery reaction: learning retained memory content triggers negative expectancy violations, beyond just demanding control
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- "Ghost of the past": identifying and resolving privacy leakage from LLM's memory through proactive user interaction
- Understanding Users' Privacy Perceptions Towards LLM's RAG-based Memory
- Assessing and Mitigating Data Memorization Risks in Fine-Tuned Large Language Models
- Relational Gains, Privacy Strains: Exploring Users' Perceptions and Experiences with ChatGPT's Memory Feature
- Privacy Leakage Overshadowed by Views of AI: A Study on Human Oversight of Privacy in Language Model Agent
- The Algorithmic Self-Portrait: Deconstructing Memory in ChatGPT
- Autonomy Reshapes How Personalization Affects Privacy Concerns and Trust in LLM Agents
- The Personalization Mirage: How LLMs Fabricate User Profiles, and Why Self-Monitoring Misleads
Original note title
users never noticed LLM memory held or inferred private information until told, then demanded proactive control over it