Understanding Users' Privacy Perceptions Towards LLM's RAG-based Memory

Abstract Large Language Models (LLMs) are increasingly integrating memory functionalities to provide personalized and context-aware interactions. However, user understanding, practices and expectations regarding these memory systems are not yet well understood. This paper presents a thematic analysis of semi-structured interviews with 18 users to explore their mental models of LLM’s Retrieval Augmented Generation (RAG)-based memory, current usage practices, perceived benefits and drawbacks, privacy concerns and expectations for future memory systems. Our findings reveal diverse and often incomplete mental models of how memory operates. While users appreciate the potential for enhanced personalization and efficiency, significant concerns exist regarding privacy, control and the accuracy of remembered information. Users express a desire for granular control over memory generation, management, usage and updating, including clear mechanisms for reviewing, editing, deleting and categorizing memories, as well as transparent insight into how memories and inferred information are used. We discuss design implications for creating more user-centric, transparent, and trustworthy LLM memory systems.

Introduction. Large Language Models (LLMs) like ChatGPT1, Gemini2, and Kimi3 are rapidly evolving from stateless tools into personalized assistants. A key technology driving this shift is the integration of memory, which allows LLMs to retain information across conversations to provide more coherent and contextually aware interactions. While this promises enhanced utility, it also introduces a significant and complex privacy challenge. By creating a persistent record of user interactions, these systems build detailed profiles that can include sensitive thoughts, personal plans, and confidential information, moving beyond transient queries to continuous user data collection. This capability creates a fundamental tension between personalization and privacy of LLM’s memory. On one hand, users desire the efficiency and tailored responses that memory enables. On the other, the opacity of how these systems remember, analyze and utilize personal data can undermine a user’s sense of control and informational self-determination. The “black box” nature of LLMs exacerbates this issue, leaving users unable to fully understand the scope of data being collected in memories or the associated privacy risks. Specifically, for RAG-based memories, they were usually extracted from users’ dialogues, as per ChatGPT, and used for latter personalized dialogues as context information. It may cause potential risks if those RAG-based memories are utilized for training or disclosed to untrusted third-party services. However, how users understand the RAG-based memories, what’s their practice and challenges remains under-explored. This study therefore investigates how users perceive RAG-based memories, manage the conflict between privacy and utility, and what’s their challenges. We aim to first understand users’ mental models of memories as their mental model largely influence whether they could are aware of the potential privacy risks and could effectively control. We then identified their current calculus around the privacy-utility tradeoffs and the protective strategies, as private memories inherently possess the conflicts between personalization and privacy leakage. We finally understand users’ challenges and expectations for improving RAG-based memory systems. Specifically, we seek to answer the following questions: RQ1. What is users’ mentral models of RAG-based LLM memories? RQ2. How do users navigate the trade-offs between utility and privacy when using or considering LLM memory, and what privacy protection strategies do they currently employ? RQ3. What are the challenges users face when attempting to align an LLM’s memory behavior with their privacy goals? We conducted semi-structured interviews with 18 Chinese participants from diverse backgrounds with varying levels of experience. To anchor the discussion in a real-world context and elicit concrete responses, the interview was centered on ChatGPT’s memory feature as a prominent case study. Towards RQ1, we find four prominent mental models of users: users regard memory as a transient, dialogue-specific buffer, as an extension of the core training data, as an active information processing mechanism, and some users explicitly acknowledged the lack of understanding. These mental models reflected different understandings and cognitive processing logics of LLMs’ memories. Towards RQ2, our research shows that users are not passive but are active agents performing a continuous privacy calculus, weighing functional benefits against perceived risks. The primary benefits involved enhance personalization and efficiency, improvec continuity and reduced redundancy, and LLM to be a better companion of assistant. The primary drawbacks include problems such as confidentiality and data leakage, profiling and unwanted persuasion, aggregation and re-identification, unauthorized secondary use, and the lack of epistemological certainty. Users employ proactive protective strategies, such as strategic privacy disclosure, proactive input obfuscation and refusal to use or workarounds. Towards RQ3, we identify an unequivocal user mandate for a fundamental shift toward user-directed memory systems built on granular control and transparency across the entire data lifecycle. This includes explicit consent at the point of generation, comprehensive management interfaces for editing and deletion, purpose limitation controls during usage, and direct agency over system-inferred information. To sum up, our contributions are: • We characterize users’ diverse mental models of LLM memory and their privacy risk perceptions. • We investigate the privacy calculus and protective strategies users employ, such as data minimization and anonymization.

• We identify the critical privacy challenges user face and distill our findings into design implications for user-centric, privacypreserving memory systems.

Related work. Research on privacy protection for LLMs follows two primary lines: privacy risk evaluation and direct defense mechanisms. Risk evaluation, which underpins any protective effort, is supported by several toolkits and benchmarks. LLM-PBE [17] supplies a structured framework for assessing risks throughout the model lifecycle, and PrivLM-Bench [16] offers a standardized benchmark to quantify data leakage. Specialized instruments such as PrivacyLens [30] evaluate compliance with privacy norms, while ProPILE [14] probes models to detect possible PII exposure. Direct defense mechanisms are engineered to intervene at different points. Some operate at the data interface: OneShield [2] filters both user inputs and model outputs, while Rescriber [50] leverages an LLM to minimize sensitive content in queries in real time. In contrast, other strategies intervene at the model level. The CPPLM paradigm [35], for example, embeds safeguards directly into the fine-tuning process to protect inference-time privacy.

Research into users’ privacy concerns in digital contexts began early, showing that users often self-regulate by selectively withholding information or avoiding services perceived as high-risk [5, 24, 26]. In the LLM domain, this manifests as a persistent trade-off between privacy, utility, and convenience [47], frequently giving rise to a “privacy paradox” in which users tolerate greater information leakage in exchange for higher utility [46]. Similarly, users of conversational agents tend to exhibit fewer privacy concerns than non-users, sharing sensitive lifestyle and health data while withholding direct identifiers [51]. These behaviors are further amplified by cognitive biases and system design. Users often rely on inaccurate mental models of LLM data flows and are vulnerable to dark patterns, which impedes their ability to grasp real privacy risks [22, 47]. Additionally, the anthropomorphic presentation of LLMs encourages oversharing, as users overestimate the system’s capabilities and attribute humanlike understanding to it [11, 33]. This tendency heightens privacy exposure and can be exploited for malicious purposes [13, 28].

The capacity to maintain conversational history is a foundational feature of LLM-driven agents, enabling coherent and contextually aware human–AI interaction [10, 20]. Early methods appended the full chat history to the model context. However, as conversations lengthened, agent performance degraded due to distraction and information loss—often called “lost in the middle” [21, 31]. Consequently, research shifted toward more sophisticated memory architectures.

Method. The study employed a qualitative approach to investigate user perceptions and expectations of memory functionalities in LLMs. We utilized semi-structured interviews to gather rich, in-depth data from participants.

We recruited 18 participants for this study through distributing questionnaires online. The participants represented a range of academic and professional backgrounds, including fields such as engineering, life sciences, social sciences, IT and design. Table 1 showed the demographics of participants. The study was approved by the Institutional Review Board (IRB) of our institution, and each participant was compensated 100 RMB for their time.

We conducted semi-structured interviews in Chinese, which allow for flexibility in exploring emergent themes while ensuring that core topics related to LLM memory were covered with each participant. The interviews focused on participants’ mental models of LLM memory, their current practices and experiences, perceived advantages and disadvantages, willingness to share different types of information, privacy concerns and their expectations of ideal LLM memory systems across its lifecycle (generation, management, usage and updating). All interviews are conducted via Tencent meeting online4, and we recorded and transcribed these for analysis.

The interview data, comprising qualitative notes and direct participant quotes, was analyzed using thematic analysis. This involved an iterative process wherein two researchers first familiarized themselves with the data. Initial codes were then inductively generated based on participants’ discussions of memory mechanisms, benefits, drawbacks, privacy risks, and desired features. Following this, a collaborative process was undertaken to discuss these initial codes and develop a unified codebook. To ensure consistency, two researchers then independently coded 20% of the data, and inter-rater reliability achieved Cohen’s Kappa of 0.90. They then independently coded the rest of the dataset. Subsequently, these codes were collated into potential themes and sub-themes, which were then reviewed, defined, and refined through further discussion to ensure they accurately captured the nuances of participants’ experiences and perspectives. The analysis specifically focused on identifying users’ mental models of LLM memories’ privacy risks, their current mitigation practices and encountered challenges, and their expectations for the design of privacy-preserving LLM memory systems across the lifecycle.

Discussion. Participants’ conceptualization of how LLM memory functions were markedly varied and often informed by inaccurate analogies. The analysis revealed four dominant themes in their mental models: (1) memory as a transient, dialogue-specific buffer, (2) memory as an extension of core training data, (3) memory as an active information processing mechanism, and (4) a widespread acknowledged lack of understanding. Mental Model 1: memory as a transient, dialogue-specific buffer. A substantial group of users conceptualized LLM memory as ephemeral, existing only within the confines of a single, continuous interaction or dialogue session. This model posits that any contextual understanding gained by the LLM is reset once a conversation window is closed, preventing memory from persisting across separate interactions. A primary misconception articulated by P1 was the belief that “the LLM’s memory can only be maintained in the same dialogue window.” This view was explicitly shared by P5, who stated that memory “exists in the same dialogue, but not across dialogues.” This mental model has direct behavioral consequences, such as users starting new chats to “reset” the LLM’s context when it becomes stuck on a flawed instruction. Mental Model 2: memory as an extension of core training data. Another prevalent mental model treats LLM memory as a mechanism that directly augments the model’s foundational training dataset. Participants with this view envision their conversations being absorbed into the LLM’s public knowledge base, akin to a continuous training process, potentially compromising their privacy. This model blurs the line between a private, personalized memory and the public, generalized knowledge of the AI. P3 wondered if memory was achieved “through training on massive data,” considering the feature a “long-term memory model ... built upon the original training data. ... comprehensively collecting new information ... then conducting further training.” This perspective was clearly echoed by P16, who questioned if memory involved “taking past chat data, re-labeling it, and then put[ting] it into its model for training”. This conceptualization often leads to heightened privacy concerns, as it implies that personal or proprietary information could be permanently integrated into the model and potentially exposed to other users. P16 later articulated Participants described both benefits and concerns related to LLM memory, reflecting a privacy calculus. This section outlines the perceived benefits (section 4.2.1) and concerns (section 4.2.2) that shaped how they evaluated this trade-off.

4.2.2 Concerns of LLM memory. Counterbalancing these benefits is a wide spectrum of privacy threats that users are concerned about. Confidentiality and data leakage: Users expressed significant concern about the unauthorized disclosure of sensitive information. This included the leakage of unpublished professional work (P1), proprietary source code (P3), and financial details like bank statements (P17). Profiling and unwanted persuasion: A primary fear was that remembered information would be used to create detailed user profiles for malicious or commercial purposes. P6 identified the risk of “exposure of personal habits and preferences, leading to targeted information, popular scams, ad calls”. P7 worried about the LLM “mastering private life activities, homework and work content, research, life and work.” Aggregation and re-identification: Users with technical backgrounds feared the power of data aggregation.

Limitations. 5.4 Limitations