People don't notice a chatbot is quietly remembering them, so the moment it reveals that profile hits like a privacy jolt.

This explores why seeing a chatbot's memory of you, meaning the profile of facts and guesses it has stored, can upset users more than other unexpected AI behavior. One caveat first: the collection has no study that ranks memory profiles against other triggers and finds them 'the strongest.' What it does have is several findings that together explain why memory is an obvious flashpoint. The first is that most people don't know the profile exists. In one study of 40 users, more than 30 had never noticed their LLM's memory feature or the private information it had inferred about them. Once they were shown it, every participant wanted to be able to edit and delete it Do users notice when LLMs infer and retain private information?. A violated expectation is sharpest when you never knew you held the expectation, and here users quietly assumed nothing was being kept.

Part of the reason is that people's assumptions about memory don't match each other or the product. Interviews with ChatGPT users found four separate mental models: memory as something that lasts only for the session, as part of the training data, as active processing, or simply as unknown What mental models do users hold about LLM memory?. Whichever model a user holds, a stored profile breaks at least some of what they assumed. It also helps to know that memory is a product decision, not a property of the model. Products choose which stored items get pulled into each conversation How do AI products decide what to remember between conversations?, so what a chatbot 'remembers' depends on design choices the user never sees.

The second factor is that the profile contains more than what you said. It contains what the system inferred about you, and those inferences go wrong in recognizably human ways. A diary study sorted chatbot memory errors into 14 types, including over-inference, mixing up who said what, and getting the user's identity wrong What types of memory errors do chatbots actually make?. Being misremembered by a machine stings in the same way being misjudged by a person does. The system also has no record of what it doesn't know about you, so it fills the gaps with confident guesses. Simply giving models a labeled list of unknowns cut hallucination roughly in half Do language models know what they don't know about users?.

The less obvious finding is that a stored profile can quietly change what the chatbot tells you, not just how it talks. When researchers set a chatbot's internal estimate of a user's socioeconomic status to 'low,' it falsely said there were no direct flights to Hawaii Can steering a chatbot's user model change its factual claims?. So seeing the profile can reveal that past answers may have been shaped by a guess about who you are. Time adds to this as well: personalization raises trust and the feeling that the chatbot is person-like, but it also raises privacy worries and each interaction lifts the baseline, so a later failure lands harder Does chatbot personalization build trust or expose privacy risks?.

What do users want instead? Not better ways to review the profile after the fact. In a test of 12 memory-control designs, people preferred controls that act at the moment something is captured, such as asking where a memory should go or setting boundaries in advance, over auditing records later Do users prefer controlling what gets remembered over auditing what was stored?. That points to a practical lesson: the shock comes less from what the profile contains than from finding out about it after the fact, and showing people more of the profile later doesn't remove that.