Do people notice on their own when an AI assistant's reply or memory has leaked their private details?
Can people detect privacy leakage in LLM agent responses without explicit prompts?
This explores whether ordinary people notice on their own when an AI agent's reply exposes private information, without anyone pointing the problem out to them first.
This explores whether people catch privacy leaks in AI agent replies on their own, before anyone flags them. The corpus says mostly no, and the reason is more interesting than carelessness. In a 300-person survey, people chose AI-drafted replies that leaked private details over their own drafts 55% of the time. The baseline was 15.7%. The researchers trace this to overtrust and to missing the breach altogether, not to people deliberately trading privacy for convenience Do people notice when AI agents leak private information?. The leaky reply often just reads as the better-written one.
The same blind spot shows up with LLM memory. In a 40-person study, more than 30 participants never noticed that their assistant's memory had stored or inferred private information about them. Once they were shown it, every participant wanted controls to edit and delete it Do users notice when LLMs infer and retain private information?. Together, these two studies suggest the problem is perception, not values. People care about privacy once they see the leak. They just don't see it unprompted. An explicit prompt or disclosure is what turns that indifference into concern.
Much leakage also happens where users never look. When reasoning models think step by step, 74.8% of the privacy leaks in their reasoning traces come from the model pulling sensitive user data into its own thinking. Longer reasoning makes this worse. Scrubbing the traces afterward hurts performance, which suggests the private data is doing real work in the reasoning Do reasoning traces actually expose private user data?. If the leak lives in a trace, a memory store, or a tool call, there's nothing in the visible reply for a person to catch.
That raises the question of whether we should rely on the model to protect privacy instead. The corpus is sobering there too. When an agent works for several parties at once, prompting, distillation and RL all move models along the same tradeoff between leaking too much and refusing too much. None of these methods escapes it Can training methods escape the leak versus over-refusal tradeoff?. Part of the difficulty may be that LLMs are weak at tracking who knows what. They look socially competent when one model plays every role, but they fail when agents hold genuinely private information Why do LLMs fail when simulating agents with private information?.
The takeaway is that neither the human reviewer nor the model is a reliable privacy filter on its own. Human review quietly fails because readers trust fluent replies, and model-side training hits a ceiling. That points toward controlling what an agent can access and remember, rather than judging each output after the fact Can a model-level filter truly contain an agent with environment access?. One gap: the corpus has no studies testing what kinds of interface cues would help people spot leaks on their own, beyond simply telling them.
Sources 6 notes
In a 300-person survey, participants preferred AI-generated replies containing privacy violations 55% of the time, up from 15.7% baseline. Researchers attribute this to overtrust and failure to notice contextual privacy breaches rather than deliberate trade-offs.
A 40-participant study found 30+ users never noticed their LLM's memory feature or inferred private information within it. Once shown the risk, all participants wanted editing and deletion controls, citing privacy as the primary concern.
74.8% of privacy leaks in language model reasoning traces result from models materializing sensitive user data during thought processes. Longer reasoning chains amplify leakage, and anonymizing traces post-hoc degrades model utility, suggesting private data functions as cognitive scaffolding.
Prompting scaffolds, distillation, and RL all move models along the same Pareto frontier without crossing it. Five experimental controls show this is structural: RL increases harm from the best checkpoint, apparent workarounds vanish under replication, and data scaling moves in the wrong direction.
Research shows LLMs perform well when one model controls all interlocutors but fail systematically when agents possess private information. This reveals that apparent social competence relies on grounding work that models skip in omniscient settings.
Show all 6 sources
A filter judges a single output at one point in time; an agent's risk spreads across memory, retrieved content, tool calls, and environmental reach. Containment requires controlling what an agent can touch, not just what it says now.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Privacy Leakage Overshadowed by Views of AI: A Study on Human Oversight of Privacy in Language Model Agent
- Leaky Thoughts: Large Reasoning Models Are Not Private Thinkers
- "Ghost of the past": identifying and resolving privacy leakage from LLM's memory through proactive user interaction
- When Are LLM Inferences Acceptable? User Reactions and Control Preferences for Inferred Personal Information
- Understanding Users' Privacy Perceptions Towards LLM's RAG-based Memory
- Assessing and Mitigating Data Memorization Risks in Fine-Tuned Large Language Models
- Whose Side Is Your Agent On? Multi-Party Principal Loyalty in LLM Agents
- Autonomy Reshapes How Personalization Affects Privacy Concerns and Trust in LLM Agents