Users' Expectations and Practices with Agent Memory

Source: Brennan Jones, Kelsey Stemmler, Emily Su, Young-Ho Kim, Anastasia Kuzminykh; CHI 2025 Extended Abstracts · 2025-04

Abstract AI agents have the potential to provide long-term personalized assistance to users, and this relies on effective long-term memory. While memory in agents has been extensively covered by prior work, there is little understanding of users’ expectations and practices with agent memory. As a preliminary investigation, we interviewed people who use AI tools with memory and analyzed online discussion posts of people’s experiences with such tools. We found that users often have incomplete mental models of how agents remember and recall information, and how their memories affect their behaviours. Users generally consider agents’ memories as belonging to different categories along a hierarchy from more generalized knowledge to more specific knowledge about the user or task. Users often desire the system’s memories to be cleanly organized by these categories. These findings reveal opportunities to design agent memory mechanisms to organize and control access to memories based on users’ task-based needs.

Introduction. AI agents, including LLM-based chatbots and tools like ChatGPT, Claude, and Gemini, are becoming more prevalent in people’s daily lives. People are also increasingly interacting with such agents over long periods of time—e.g., using them as personal AI assistants (e.g., [2]) and ‘coaches’ for wellness, exercise, life goals, and work productivity.

Such AI agents have the potential to provide always-available personalized and contextualized assistance to users, and to work in collaborative ‘complementarity’ with the user. However, in order to achieve this, the user and the agent can benefit from achieving co-understanding of each other. One aspect that contributes to co-understanding is memory. To achieve sufficient co-understanding in longer-term interactions between humans and agents, the agent needs to remember the right things about the user and the interactions over time, and refer to those memories optimally in giving effective assistance to the user. Further, it is also helpful for the user to have sufficient understanding of the agent’s decision-making processes and how its memories and knowledge impact its behaviours.

Memory in LLM-based agents has been extensively covered by prior research (e.g., [11, 36]). Strategies for maintaining an agent’s memory include summarization (e.g., [29, 32]) and refinement (e.g., [37]) of conversational and interactional histories, maintaining vector databases and retrieving relevant conversations or information through semantic or linguistic matching (e.g., [13, 24, 38]), maintaining key-value pairs in traditional databases (e.g., [21]), or maintaining lists of ‘facts’ or ‘observations’ learned from user interactions (e.g., [1, 5, 24, 33]). There are also systems that allow users to influence or modify agents’ memories through, for example, direct manipulation (e.g., [15, 34]). Some systems’ memory mechanisms (e.g., [18]) are designed to mimic existing psychological models of human memory (e.g., [4]), while others (e.g., [21]) are designed to mimic existing technological metaphors such as traditional operating system memory. However, there is a lack of understanding of users’ expectations of AI systems’ memory mechanisms while using such systems over longer periods of time. Furthermore, there is a lack of understanding of users’ practices of managing these systems’ memories. Achieving such understanding could help designers build memory mechanisms for agents that involve users, consider their mental models (in line with), and keep the human in-the-loop. To achieve this understanding, we conducted interviews with six participants who use personalized AI tools with long-term memory on a regular basis, and thematically coded these interviews to explore users’ expectations and mental models of such systems’ memory mechanisms and their practices with managing memory in these systems. We also conducted a thematic analysis of publicly available posts and discussion threads on Reddit, in which people discussed their experiences using the memory features of personalized AI systems. Our findings illustrate that, in general, users often have an incomplete understanding of how existing agents remember and recall information from users’ interactions. Users generally think of the system’s memories as belonging to different categories that can be organized along a hierarchy from more generalized knowledge (e.g., factual memories of general knowledge facts, usually contained in the model’s training data or fetched externally through retrieval-augmented generation, or RAG), to knowledge about the user (e.g., their personal or social details, preferences, etc.), to more specific knowledge about the domains, projects, or tasks that the agent assists the user with. Users’ needs for the system to store and retrieve memories along the different positions of this hierarchy evolve depending on the task or stage of the activity they are working on. Our findings also reveal users’ existing approaches to organizing, separating, and abstracting access to memories across different tasks, projects, and domains, including their use and organization of chat threads, customs prompts and instructions for defining personas and roles, fine-tuning models, and using different user accounts. These findings reveal opportunities to design agent memory mechanisms to organize and control access to memories via hierarchical structures based on users’ task-based needs.

Method. We conducted a study with the goal of better understanding users’ expectations of AI systems’ memory mechanisms and their practices of managing these systems’ memories while using them over longer periods of time. Our data was collected from two sources: (1) one-on-one interviews with participants who use personalized AI tools with memory, and (2) online posts and discussions threads on Reddit, where people discussed their experiences using personalized AI tools with memory. Our research questions were as follows:

(1) What are people’s mental models and expectations of what AI memory systems do? For example, what does the user expect the AI system to remember? How does the user expect the AI to behave when recalling old memories? How do people expect their information to be abstracted when stored in memory? (2) What are people’s unsatisfied needs with AI memory systems? How do these unsatisfied needs affect users’ interactions with and perceptions of the AI system? (3) What are people’s current practices with managing and exploring the memory of a personalized AI system?

For example, how does the user (try to) instruct or get the AI to remember, recall, or forget something? When and why does the user want to manage the system’s memories?

We conducted semi-structured one-on-one interviews (see Appendix A for the interview guide) with six participants (each identified as ‘P#’ in the findings) who use personalized AI tools with long-term memory on a regular basis. Each interview lasted between 30 and 45 minutes. The interviews were conducted on Zoom, and each participant was reimbursed with CAD$20. The interviews were recorded and transcribed for analysis. Participants also filled out a brief survey (see Appendix B) where they provided basic demographic information and answered questions about their current and prior use of AI tools. Four of our participants identified as female and two as male, ranging from 24 to 35 years old (M = 29, SD = 5). Three participants expressed that they use LLMs or personalized AI tools at least once per day, while the other three expressed that they use these tools at least once per week. Of the specific tools that participants mentioned using on a regular basis, all of our participants expressed using ChatGPT, while three also use Claude, two also use the Microsoft Copilot chatbot, and one each also use Perplexity, NotebookLM, Grammarly’s AI features, and the Cursor AI code editor for code completion and chat. Our participants expressed using these tools for activities such as “productivity tasks, writing assistance, usability analysis,” “analyzing qualitative data,” “email writing and editing,” “code-related questions, brainstorming, error troubleshooting,” “learning something from scratch,” “conversational [web] search[ing],” and “as a personal tutor, search engine, and life companion.” All six participants rated their “level of expertise with using these tools” as 4/5 (with 1 = ‘beginner/novice’ and 5 = ‘advanced/expert’). However, when asked about their “level of technical knowledge of LLMs” (i.e., their “technical understanding of how LLMs work” and their “technical understanding of memory in LLMs and LLM-based tools and agents”), four participants rated themselves as 4/5, and one participant each rated themselves as 2/5 and 1/5 (with 1 = ‘beginner/novice’ and 5 = ‘advanced/expert’). We also collected and performed a preliminary analysis of 54 publicly available discussion threads on Reddit, in which people discussed their experiences using the memory features of AI systems. The threads were collected systematically from Reddit communities centred around topics such as AI tools, agents, LLMs, generative AI, and machine learning. In addition, relevant threads were collected from general Q&A Reddit communities where people asked questions related to the memory of AI systems. Discussion threads were collected from these communities using search terms and keywords such as “memory,” “forgetting,” “remembering,” “chat history,” “summarization,” “temporary chat,” and “personalization.”

Discussion. Users’ Perceptions and Desires for Transparency Overall, we found that users often have an incomplete understanding of AI systems’ memory mechanisms, including what the system remembers, how the system decides what information to store in its memory, and how the system’s memories influence its behaviours or outputs. As a result, our participants expressed their desire for the system to provide greater transparency of what it remembers about the user and its previous and ongoing interactions with them.

“When I’m conversing in it with some other ways, or, you know, just regular email writing or something, it’ll also sometimes say ‘memory updated’. And then, but, I think I don’t really know how it decides what is important information to remember versus what is not important information to remember in, like the prompts that I write.” — P3 “Definitely not [transparent]. Like, I’m... I can never say for sure whether it remembers this thing or not. And even sometimes I tell it to remember, it may forget.” — P6 Users also want the system to provide greater transparency of how its past memories influence its present behaviours:

“It would be interesting, like I mentioned, to know which parts of the memory it has about me go into, like, which responses, and to understand more about how it decides which portions [...] of that list [of memories] is like... you know, [it is] 100 or 200 items... like, that’s a lot of things to remember about me. And to decide, like, which parts of that list actually go into a response versus not.” — P3 Some of our participants were surprised about how much the systems they use remember. For instance, our participants who have used ChatGPT, Copilot, and other LLM-based chatbots expressed that they were surprised about how much the system remembers sensitive or personal information about them.

“I didn’t notice it, but now I’m looking at the memory, it’s definitely collected a bunch of information about me. [...] It’s scary.” — P1 “Yeah, it remembers quite a lot. Oh, my God.” — P2 Participants were particularly surprised that the systems they use preserve memories containing details that they consider to be ‘small’, ‘mundane’, or not directly related to the activities that they use them for.

“So it saved my supervisor[’s name]. Which makes sense. Because it read emails, like, I polish my emails to my supervisor with GPT, so it knows. And the really scary one is, it has my student ID. And then my employee ID. I think I don’t put [those] there a lot. Maybe it’s just... for the one or two times I’m trying to contact the department, like for support, where I put my information there [in the email], and so I’m really surprised. Like, I didn’t keep putting it there, but [ChatGPT’s memory] captured it.” — P6 In some cases, the systems were able to use these small details to make detailed inferences about the user—for example, about their personality.

“I actually tried asking ChatGPT to guess my MBTI [Myers–Briggs Type Indicator personality test]. And [...] it got three out of the four [indicators] correct. So like that, that kind of threw me off, because I don’t talk much about my personality. Like I mentioned, I’m using [ChatGPT] mostly for productivity. I use it to help me, you know, write papers, and then, to, you know, improve [the] language of emails, and things like that. [...] And then ChatGPT guessed [my personality], just based on regular interactions that were mostly about work and research. [...] So I think that was a little, where I was caught off guard.” — P3 While some were initially amused by this, most participants did not want the system to remember such personal details over a longer term.

Conclusion. This work provides preliminary understanding of users’ long-term expectations and needs from AI systems’ memory mechanisms, their practices of managing these systems’ memories, and how their needs and practices evolve over time based on the stages of the activities they work on with the agents they use. To expand on these findings, we plan to run further studies (e.g., more in-depth interviews and longer-term studies with more participants) to better understand people’s evolving needs over a longer period of time. We are also interested in understanding how certain design choices in AI systems (e.g., anthropomorphic design traits, direct manipulation vs. interface agents) affect people’s perceptions of and practices toward agent memory.