When an AI assistant quietly decides what to remember about you, what rules should govern that memory?

This explores the privacy rules that should apply when an AI assistant decides by itself what to remember about you. Start with how often that happens. In one analysis of 2,050 real ChatGPT memory entries, the system created 96% of them without being asked. Users explicitly requested only 4% Who actually controls what ChatGPT remembers about you?. So unrequested memory isn't an edge case. It's how the feature mostly works. The content matters too: 52% of those entries recorded psychological insights about how a user thinks or feels, and only 28% held what the law defines as personal data What kind of information does ChatGPT actually remember about users?. Many privacy rules are built to protect identifiers like names, addresses and account numbers. They largely miss the category these systems store most: inferences about who you are.

The first safeguard the research points to is simply making these memories visible. In a 40-person study, more than 30 participants never noticed their assistant's memory feature or the private details it had inferred. Once they were shown, every participant wanted to edit and delete Do users notice when LLMs infer and retain private information?. Part of the problem is that people hold very different beliefs about what memory even is. Some think it lasts only for a session, some think it feeds into training, and some have no idea What mental models do users hold about LLM memory?. You can't consent to something you picture wrongly. When users are asked what they want, they describe control at every stage: consent when a memory is created, a place to review and edit memories, limits on what memories can be used for, and specific say over inferred data What memory controls do users actually want from LLMs?. This fits what ChatGPT users worry about most: what gets collected in the first place, ahead of how it's used or how long it's kept What privacy concerns matter most to ChatGPT users?. For unrequested memories, the moment of creation is the natural place for a safeguard.

Two findings complicate the obvious fix of adding an edit screen. First, feeling protected and being protected can come apart. In a study of 188 people cleaning up chatbot transcripts, those who edited by hand felt just as confident about their privacy as those who used AI suggestions, yet they left far more identifying information behind Why do people feel safe sharing when they haven't actually removed sensitive information?. A memory dashboard could create the same false comfort. One proposed answer is to split the work: users state what they want kept private, and the AI finds and edits the matching memories. This could lower the effort involved, but it hasn't been validated yet Does splitting privacy work between users and AI reduce cognitive burden?. Second, deleting a memory doesn't reliably make it gone. A benchmark of shared agent memory found that retrieval-based and vector-store systems kept leaking information after it had been deleted. No current system handled usefulness, access control and real forgetting all at once Can shared agent memory systems reliably delete information?. Private details can also resurface in a model's reasoning steps, which is where most leaks in one study came from, and scrubbing those steps afterward made the model less useful Do reasoning traces actually expose private user data?. Taken together, a delete button needs proof that deletion worked. Cryptographic commitments, which let a system prove what happened without revealing the content, are one possible direction, though they bring their own questions about who keeps the underlying data Can commitments protect sensitive agent data while enabling verification?.

There is also a reason the stakes grow over time. Long-term studies show that personalization raises trust and makes people treat the chatbot more like a person, while privacy concerns grow alongside Does chatbot personalization build trust or expose privacy risks?. The more an assistant seems to know you, the more you tend to tell it, and the more it can infer without being asked. Here is what the research suggests that you might not have expected. The main issue with unrequested memory may be less about secrets leaking and more about a profile of your psychology being built that you never see, which data law barely recognizes and current deletion methods can't fully erase. The corpus is stronger at identifying these problems than at proposing tested solutions. Notice-at-creation, purpose limits on inferences and verifiable deletion are all supported by the evidence, but none has yet been shown to work in a deployed system.