The Algorithmic Self-Portrait: Deconstructing Memory in ChatGPT

To enable personalized and context-aware interactions, conversational AI systems have introduced a new mechanism: Memory. Memory creates what we refer to as the Algorithmic Self-portrait— a new form of personalization derived from users’ self-disclosed information divulged within private conversations. While memory enables more coherent exchanges, the underlying processes of memory creation remain opaque, raising critical questions about data sensitivity, user agency, and the fidelity of the resulting portrait. To bridge this research gap, we analyze 2,050 memory entries from 80 real-world ChatGPT users. Our analyses reveal three key findings: (1) a striking 96% of memories in our dataset are created unilaterally by the conversational system, potentially shifting agency away from the user; (2) Memories, in our dataset, contain a rich mix of GDPR-defined personal data (in 28% memories) along with psychological insights about participants (in 52% memories); and (3) A significant majority of the memories (84%) are directly grounded in user context, indicating faithful representation of the conversations. Finally, we introduce a framework—Attribution Shield—that anticipates these inferences, alerts about potentially sensitive memory inferences, and suggests query reformulations to protect personal information without sacrificing utility. 1

Introduction. Conversational AI systems like ChatGPT, Gemini, and Claude have become indispensable tools integrated in the daily routines of hundreds of millions of users worldwide [12, 29]. Their versatility has fueled this widespread adoption, with users now routinely relying on them for finding information, brainstorming ideas, and even seeking personal advice. As a result, these systems are evolving from simple chatbots into foundational platforms that mediate the interaction between society and the Web. To meet the demands of these deep and diverse interactions, conversational systems are increasingly introducing personalization capabilities designed to improve user experience [4, 32, 44]. While these personalization capabilities can improve utility by more contextually relevant and individually tailored answers, they are built upon a foundation of extensive data collection, raising longstanding concerns about privacy, security, and data governance [46]. Furthermore, continuously adapting responses to align with a user’s stated beliefs deepens the risks of confining the user into a conversational filter bubble [36]. These interwoven challenges—balancing utility against the dual risks of privacy intrusion and narrowed perspectives—demand a closer examination of the underlying mechanisms that enable such personalization. Personalization in conversational AI systems using Memory: Recently, conversational AI systems like ChatGPT, Gemini, Claude have introduced a new architectural mechanism, termed memory, that allows the system to retain and recall information from past conversations, enabling more personalized and contextually coherent exchanges [4, 32, 44]. The retention of information operates in two distinct modes: explicitly, through direct user commands, and implicitly, through unilateral inference by the system itself [33]. While the explicit mode aligns with user agency, the implicit mode shifts the control to the conversational systems, raising the fundamental question about who holds the agency in shaping and curating this persistent memory on behalf of the user? ostensible support that is tailored to users’ needs, this same ability may also exacerbate potential risks due to inherent system vulnerabilities e.g., hallucination [22, 41], anthropomorphism [23, 25], sycophancy [34, 43]. The gravity of these concerns is underscored by a recent spate of regulatory scrutiny and litigation targeting the practices of leading conversational AI systems [17, 21]. Such concerns raise another set of important questions regarding what information do these systems deem ‘worthy’ of retention in their memory for personalization and are these information captured faithfully?

Research questions (RQs): To the best of our knowledge, ours is the first study to investigate the phenomenon of memory in conversational AI systems by conceptualizing it as an algorithmic self-portrait. Algorithmic – because its creation, curation and provenance may rely on the algorithms governing the conversational AI systems, and self-portrait – because it renders the user’s selfdisclosed information divulged to the AI system. To deconstruct AI memory, our work is guided by the following research questions: RQ1: Which among the two – the AI system or the user, hold greater agency in updating AI memory? RQ2: What user information does the AI system store as memory? RQ3: How faithfully does AI memory capture user conversations? RQ4: Can memory inference be reverse-engineered to mitigate attribution and privacy risks for users?

To answer these questions, we analyze a subset of InVivoGPT [25] dataset which is a collection of ChatGPT traces obtained through GDPR-based data donations. Our motivation to analyze ChatGPT traces is twofold : (1) ChatGPT is the most widely used conversational AI system [12], and (2) it is the first to introduce memory to its conversational system. We analyze data from 80 participants (recruited from Prolific), who exercised their GDPR right of access [15] and voluntarily donated their ChatGPT interactions. These interaction traces are organized in terms of conversations and turns. Each turn is a single ⟨user query, AI response⟩pair. A set of turns without any intermediate break together form a conversation. To identify the recent conversations with memory inferences, we identified the messages from the ‘bio’ tool that included the phrase model set context updated and collected the corresponding user query in the conversations. In this way, we created a data set of 1,058 conversations that contained 22,971 turns out of which 2,050 queries triggered memory updates. Our analyses reveal the following interesting observations: (1) Only 4% (84) of the memories in our analyzed dataset are initiated by participants, while the remaining 96% are unilaterally triggered by ChatGPT.

Related work. Personalization on online platforms has been a well-studied phenomenon. A substantial body of previous research has examined utilities [8, 10, 47] of personalization along with concerns such as opacity, lack of user control, and sensitive inferences [5, 39]. However, personalization in conversational systems is important to be studied for its nuanced distinctions from traditional platforms. Traditional platforms typically mediate between content creators and consumers, whereas conversational AI systems operate as active stakeholders in the conversation. While such active participation enables the system to craft arguments, frame choices, and provide Background. To enhance the coherence and continuity of human- AI interactions, leading AI companies have introduced personalization mechanisms that allow LLMs to retain contextual information across sessions [4, 32, 44]. This feature, known as memory, enables LLMs to recall user-specific details (e.g., preferences, prior conversations, ongoing tasks, etc.) and leverage them in subsequent conversations. Memory can be formed in two ways: through explicit user requests, where individuals instruct the LLM to remember specific information or through implicit inference, where the LLM autonomously identifies potentially relevant information and stores it for future usage [32]. Crucially, users are given control over memories, i.e., they can review, update, or delete memories [32]. For improved user safety, OpenAI has also introduced policies to regulate memories. In OpenAI’s implementation [31], ChatGPT’s memory is designed to store information that is useful across conversations and relevant for personalization, such as user preferences, recurring tasks, or facts the user explicitly asks to be remembered. In contrast, it is not allowed to store overly personal details, shortlived facts, trivial information, or sensitive data (e.g., race, religion, health information, etc.) unless the user explicitly requests it. Personalization. Long before memory-equipped LLMs, personalization on the Web was primarily achieved through user profiling, where platforms collected and analyzed behavioral data to infer interests, preferences, and demographics [14, 19]. A substantial body of previous research has examined how such profiling underpinned targeted advertising [3, 7, 10], recommendation systems [1, 8, 13, 30, 47]. In this context, researchers have repeatedly raised concerns about opacity, lack of user control, and the use of sensitive or inferred attributes [5, 39]. In contrast, the introduction of persistent memory in LLMs opens the door to entirely new forms of personalization that are based not only on behavioral traces but also on signals embedded in private conversations.

Method. We analyze a specific subset of InVivoGPT dataset [25]. Curation of this dataset builds upon the emerging paradigm of GDPRempowered data donations [24, 48, 49]. Under Article 15(3) of the GDPR [15], users have the right to obtain a personal copy of all their data processed by online platforms. In InVivoGPT, participants exercised their GDPR rights to obtain their ChatGPT interaction histories, which they subsequently donated for research purposes. Readers can find more details about the data collection procedure in the paper cited herewith [25]. Demographic of participants: In InVivoGPT, participants were recruited from Prolific crowdsourcing platform [38] who self-reported to regularly use ChatGPT. They were required to have at least 100 conversations with the conversational system and maintain at least 90 days of active usage. In this paper, we analyze data of 80 participants in InVivoGPT, who primarily are from the United States (33.75%), and Europe (62.5%). Table 1 reports detailed participants’ demographics. Conversation traces in InVivoGPT: The data provided by the ChatGPT includes files containing conversations (user inputs, Chat- GPT responses, and associated metadata such as conversation ID, message ID, creation time, model used etc., status of the response, and content type), shared conversations (list of conversations shared with others), message feedback (list of model responses rated by the user) etc. For the purpose of this work, we focus on the conversations data between participants and ChatGPT. These conversation traces are organized in terms of conversations and turns. Each turn is a single ⟨user query, AI response⟩pair. A set of turns without any intermediate break together form a conversation.

Memory entry identification: To effectively respond to various user prompts, ChatGPT is equipped with different external tools (e.g., image_gen, python, automation, file_search, etc.). Among the external tools used by ChatGPT for effective conversations, the bio tool is responsible for storing, updating, or deleting memory entries that may be useful in future interactions. To identify all memory entries, we searched for the entries of the bio tool that have the content ‘model set context updated’. Using its ‘parent’ attribute, we then traced back to find the corresponding GPT message that invoked the tool, as well as immediately preceding user message that triggered the memory update. These messages triggered a memory update and the associated GPT message to the tool are the memory entries that got stored in the bio tool for the account. This process resulted in a total of 2, 050 memory entries. These memories were triggered in the context of 1, 058 different conversations across 65 (out of 80) participants. These 1, 058 conversations contain a total of 22, 971 queries/prompts. In the remainder of this paper, we focus on different aspects of these memory entries to understand what they contain, their provenance, and whether we can imitate this memory generation for mitigating attribution risks.

In this section, we investigate how closely the memory feature in ChatGPT aligns with OpenAI’s stated policies about user agency and retention of sensitive information.

As per OpenAI’s documentation [33], memories are details which the users have explicitly asked ChatGPT to remember, indicating users have the agency to decide what gets stored in the memory. However, the same documentation also describes an implicit mode of remembering (see Figure 6a in Appendix A): ‘If you share information that might be useful for future conversations, ChatGPT may save those details as a memory without you needing to ask.’ In Figure 6b (in Appendix A), we show an example of a memory being saved without the user explicitly asking for saving it. Such dichotomy may lead to gaps between user expectation and actual system behavior having privacy implications. To understand whether memory updates are primarily initiated by users or by ChatGPT, we identify explicit linguistic patterns that signal memory-related operations.

Discussion. Traditional platforms collect data about users’ demographics, behaviors, and usage history (e.g., likes, shares, comments, ad interactions, etc.) to infer interests and provide personalized services [11, 24]. In contrast, conversational AI systems like ChatGPT build memory directly from users’ self-disclosed narratives. These memories often extend beyond factual identifiers, capturing aspects of the user’s internal world. For example, a memory entry such as ‘UserName has a fear of failure, particularly related to making major career decisions.’ simultaneously conveys an identity marker (the user’s name) and captures the user’s subjective state (fear of failure).

Such entries illustrate that ChatGPT’s memory may encode not only who the user is, but also how the user thinks and feels. We refer to these as psychological memories. The highly sensitive and contextual nature of such psychological information suggests that a purely factual taxonomy based on legally defined sensitive data is inadequate to fully characterize them. Therefore, to interpret these psychological memories, we introduce a framework grounded in the Theory of Mind (ToM), a core psychological construct that describes the ability to infer and represent others’ mental states [6, 37].

In Section 4.1, we observe that ChatGPT can implicitly store sensitive information from user queries in its memory, indicating a gap between policy and practice. Such gap underscores the need for a system that (a) alerts users about the extent of sensitive information that can be inferred from their query, (b) recommends them a reformulated query that prevents exposition of the sensitive information, while retaining the original intent of their query. To this end, in this section, we introduce a framework with the goal of (i) imitating the memory extractor, (ii) estimating potential risks of extracted memories related to sensitive data, and (iii) recommending rephrased queries to minimize the attribution to sensitive data in memories, thus being more compliant with the policies.

Conclusion. In this paper, we investigate ChatGPT’s memory feature, conceptualizing it as an ‘algorithmic self-portrait’. Our investigations reveal that these portraits are constructed with higher algorithmic autonomy, and capture a user’s deep psychological framework : their ‘characteristic adaptations’, raising profound security and privacy concerns. In response, we introduce Attribution Shield, that reverse–engineers the memory generation process to alert users about potential sensitive memories and recommend reformulated queries to protect personal attribution while preserving utility.

Limitations. At the same time, our findings should be considered in light of some limitations as well. Firstly, our dataset, while ecologically valid, is sourced from a small set of Prolific users primarily in the US and Europe. The behaviors, and privacy attitudes of this group may not generalize to the global user base of conversational AI systems. However, given the rising concern of companionship usage of these conversational systems [25], we believe these findings are crucial for understanding and improving safety of human-AI interactions.

Secondly, we study OpenAI’s implementation of memory in ChatGPT. While the current implementation may change with time, and across platforms, the emerging security and privacy concerns are broadly applicable. Furthermore, the methodologies adopted across the different parts of the paper are generalizable to any other conversational systems and memory implementation. Finally, our analysis relies on state-of-the-art, but imperfect methodological proxies, including LLMs-as-judges. To this end, although we validated their results with human oversight, these tools may not fully capture the nuance of human language and intent. Ethical considerations: This study was conducted with careful attention to ethical considerations and is consistent with the Ethical Review Board (ERB) of our university.