"Ghost of the past": identifying and resolving privacy leakage from LLM's memory through proactive user interaction

Introduction. The widespread use of Conversational Agents (CAs) based on Large Language Models (LLMs) has facilitated natural language communication while simultaneously posing significant challenges to users’ privacy [24]. To enhance user experience and minimize repeated input of task-related background information, LLM service providers, such

Related work. With a service targeting conversational assistants, we detail privacy challenges in LLM-based CAs memorization. To optimize conversational performance, LLMs inherently require vast amounts of data for their training, often encompassing user interaction data [62]. However, a side effect of LLMs is the unintentional memorization of the training data, which also contain user input data, including personally identifiable information (PII) [12, 66], which might also be included in the generated output. For example, ChatGPT, even with safety precautions, can inadvertently disclose PII through specifically crafted prompts. Users engage with LLM-based CAs through natural language, which is traditionally reserved for humanto-human communication. This can lead them to perceive these agents as human-like. Studies suggested that anthropomorphizing can increase user information disclosure [34, 41]. Anthropomorphizing can inflate users’ perceptions of the CAs’ competencies, fostering undue confidence, trust, or expectations in these agents [41, 100]. With more trust, users might be more inclined to share private information, even in contexts typically associated with sensitive personal information [41, 80, 100]. Anthropomorphization may amplify the risks of users yielding effective control by trusting CAs unquestioningly. Moreover, more private information may be revealed when CAs leverage psychological effects, such as nudging or framing [84]. Generalized to AI technologies, the past literature [42] classified the risks as invasion risks [55, 61, 67, 73], data collection risks [73], data dissemination risks [4, 13, 44] and data processing risks [64, 65, 72, 73, 77]. Invasion risks encompass a range of activities that intrude upon an individual’s personal space or solitude. Intrusion risks encompass actions that disturb one’s solitude in physical space [73], which include personalized ads. Besides, With the increasing complexity of human-AI interaction [87] and the tasks [2], memory becomes significant during the interaction [5, 31]. Current LLMs remain opaque about the usage of memory [31] and the context [85, 94]. To address these challenges, researchers have explored various strategies to manage memory more effectively and transparently. Some proposed techniques for retaining the persona of chatbots [45, 91]. Other methods guarantee the responses generated are contextually appropriate, such as summarizing [81] and refinement [95], aiming to minimize redundancy while maintaining essential information. Relevant memories can be retrieved utilizing information retrieval techniques to contextualize current inputs to AI [5, 86].

Method. We first revealed the cognitive gap of participants, then presented users’ expectations towards future improvement. We detailed three findings shortened as F1 to F3.

3.5.1 F1: Memory Mechanism is Opaque. The process of memory generation within the system is problematic due to its opacity. Without explicit notification, 30/40 of users were un-aware of the notification when memory and context were added. The memory was also showed in the management interface in a plain manner, without highlighting the potential private information, resulting in no participant proactively noticing the privacy inside memories. The past input were even not showed in a structured panel, leaving no participant aware of the potential privacy leakage. Although all participants re-vealed private information could be directly contained in the past input or memory during generation, 37/40 participants also envisioned private information could be inferred from multiple past inputs or memories. Worse still, 28/40 users were totally un-aware of the memory usage. Even these 28 participants who were aware of the memory usage, they could not understand or guess out how the memories are leveraged and integrated. Participants also has no manner to effectively regulate or modify the memory. The management panel is hard to find, and 35/40 participants never clicked it. The 5 participants who clicked it exhibited simple behaviors. Three participants directly deleted all the memories because of the memory risk, whereas two participants chose to retain all the memories regardless of the privacy risk. Participants typically commented, “I never considered GPT has this function before this day. I am definitely not known about the private information inferred from the memory.”

3.5.2 F2: Users Lack the Privacy Awareness towards the Memory Mechanism. Despite the fact that users were un-aware of the long-term memory generation, they were also un-aware of the private information contained or potentially inferred from the past input and the memory unless explicitly told. 30/40 participants would never notice the private information in the past input before the experiment, and unfortunately, 36/40 participants never notice the private information in the memories before the experiment. They were even more un-aware of the risk that the private information could be inferred through combining different past user input and memory, which was exactly done in the usage of the long-term memory. In fact, among the interviews, all participants echoed “I have never thought about these information usage and leakage patterns before.” All participants were never aware of the private information inferred combining the past inputs and memories. However, interestingly, participants could understand the inference process after explicitly told. This further un-veiled the in-transparency of the system’s memory management.

3.5.3 F3: Users Need the Control of the Memory Mechanism. All participants reflected they lacked of control over the memory mechanism, highlighting the necessity for proactive memory management. All participants expressed the need for greater autonomy in controlling their memory, which can be categorized into three key actions: editing, adding, or deleting memories. Notably, a considerable percentage of participants identified privacy as the primary motivator for managing their memories, with privacy concerns being the most critical factor (35/40), followed by the accuracy of the stored information (14/40). 36/40 participants commented “I definitely need the system to provide me with the proactive modification permission.” A few participants also hoped besides proactive modification, the system could automatically help them manage private information. When it comes to private information, users indicated the need to make decisions about deleting such data based on the usage, importance, and potential privacy risks associated with the memory.

Discussion. 6.2 Designing and Managing Private Information in Memories Memories are essential in human-AI interactions, enhancing personalization by retaining past exchanges. While the risks of using memory data for training are similar to those of using direct user inputs, the threats are more severe because memory data is retained longer, increasing the potential for misuse. We further found users seldom noticed and cared the existence of memory (see Section 3.5). This paper presents the first framework for managing private information in AI memories, specifically addressing vulnerabilities to membership inference attacks [63] and privacy leakage [76]. Previous research [88] has addressed memory usage in AI systems but has overlooked the private information within memories and failed to model users’ mental constructs regarding privacy. Recognizing that not all memory information is essential, we propose a collaborative method involving both users and large language models to select, retain, and utilize memory data according to users’ preference [3]. This approach aims to optimize the balance between leveraging valuable memory information and safeguarding user privacy.

6.4 Design Implications We proposed several design implications for the memory management of LLMs to fertilize future design. Balancing Control and Efficiency in Private Memory Management Through Proactiveness Levels. We used hierarchical design to balance the control and efficiency, where users could click and collapse to view the original source of private information inference result. This allowed the completion of main task while maintaining efficiency for privacy protection. Hierarchical design in privacy protection could also be leveraged in privacy and security settings on smartphones [10] and text anonymization [78], where multiple levels of information or information source needed to be demonstrated efficiently and in the same time reduce users’ mental load. Visualizing the Source of the Privacy Risk to Foster Privacy Awareness. MemoAnalyzer enhance user privacy awareness by visually displaying how inferences are made from stored memories, which similar systems in online chatting [76] and personalization [3] scenarios could also adopt. A clear, intuitive visualization of the data relationships and inferences, perhaps through highlighting key phrases or linking them to past user inputs, will help users understand how their personal information is being processed. This transparency can promote informed decision-making regarding which memories or information to retain, modify, or delete, ultimately fostering greater trust in the system’s privacy protections.

Assigning Distinct Roles to Users and AI to Enhance Agency and Efficiency. Assigning users and AI different roles in the privacy management process can significantly improve both perceived agency and efficiency. For instance, users could focus on indicating their preferences or highlighting sensitive information [3], while the AI executes memory adjustments such as deletion, modification, or preservation based on those preferences. This role division reduces the cognitive burden on users, allowing them to concentrate on higher-level privacy decisions while ensuring the system efficiently handles the operational aspects of memory management.

Conclusion. This paper presents MemoAnalyzer, a proactive memory management system designed to mitigate privacy risks in human-LLM interactions. Our research highlights the opaque nature of memory mechanisms in current LLMs, which users are largely unaware of. Through a semi-structured interview and a five-day user study, we identified a significant gap in user awareness and control over long-term memory retention. MemoAnalyzer effectively addresses these issues by providing transparency, visualization, and user-driven control over private information, which was validated through improvements in privacy awareness, perceived control, and user satisfaction. This work contributes to the broader discourse on privacy-conscious AI design by demonstrating how user-centric privacy tools can enhance trust and control without impacting system performance. Future work can explore the scalability of MemoAnalyzer in more diverse real-world settings and investigate further improvements in user interaction with privacy management systems.

Limitations. We acknowledged that our study have limitations and regard these as future directions. First, although we tried to diversify the background of the participants, we recruited the participants through snowball sampling in the campus, which restricted the age and educational background. University students was a group with relatively higher educational background than the average [8] and may understand the memory mechanism easier. This would further highlight the problems that memory mechanism is opaque and hard to understand. Second, the experiment may face social desirability [17] and recall bias [18] of the participants, during which participants may utter more opinions towards memory mechanism. We selected the most common problems of participants and envisioned these were real problems that needed handling.