Chatbot users imagine totally different versions of what the AI remembers about them — and act accordingly.
How do different mental models of LLM memory affect user expectations?
This explores how people's beliefs about what an AI chatbot remembers, and how it remembers, shape what they expect from it and how careful they are with it, and how those beliefs compare with how LLM memory actually works.
This explores how people's beliefs about what a chatbot remembers shape what they expect from it, and whether those beliefs match how LLM memory really works. The corpus has one study that tackles the question head-on, and it's a striking one. In interviews with 18 ChatGPT users, researchers found four mental models that don't fit together. Some users think memory is a buffer that empties when the session ends. Some think it's an extension of the training data, so what you type becomes part of the model. Some think it's an active process that sorts and reuses what you share. Some simply don't know What mental models do users hold about LLM memory?. These aren't harmless differences in vocabulary. The study found that each model leads to different privacy awareness and different behavior. If you think the slate wipes clean, you'll share freely. If you think your words feed the model, you'll hold back.
What makes this more interesting is that all four models are partly right, because engineers really are building memory in very different ways. One approach, ReadAgent, works the way a person reads a long book. It compresses a document into short 'gist' summaries and goes back to the full text only when it needs a detail Can LLMs read long documents like humans do?. That's close to the 'active processing' model. Another approach, MeMo, trains a separate memory model that holds new knowledge while the main LLM stays untouched Can a separate memory model inject knowledge without touching the LLM?. That breaks the 'training-data extension' model: knowledge can be added to a system without changing the model's own weights. So no single picture is correct. The right one depends on which product you're using, and products rarely tell you.
The less obvious lesson concerns expectations about quality, not privacy. Most mental models assume that if the AI remembers the right things accurately, its answers will get better. MemTrapBench challenges that. All five memory frameworks it tested did worse than using no memory at all, by more than 10%, even though the stored memories were accurate and relevant to the task Can relevant memories actually harm LLM reasoning?. The failure happens when the model uses the memory, not when it stores it. A user who expects 'it remembers me, so it'll understand me better' may get the opposite: earlier context pulling the model's reasoning off course.
Taken together, users worry mostly about what memory exposes, while the research suggests they should also wonder what memory distorts. The corpus is thin here, though. Only one study looks directly at user mental models, and none yet tests whether a better mental model leads to better outcomes. Treat this as an open question rather than a settled one.
Sources 4 notes
Semi-structured interviews with 18 Chinese ChatGPT users revealed four distinct mental models: memory as session-only buffer, as training-data extension, as active information processing, or as simply unknown. These models directly influence privacy awareness and user behavior.
ReadAgent compresses documents into gist memories before knowing the task, then retrieves details only when needed, extending effective context 3–20× and outperforming retrieval baselines on long-document QA.
MeMo trains a dedicated memory model to encode new knowledge, eliminating inference-time search costs that scale with corpus size. It avoids fine-tuning risks and works with frozen proprietary models, but trades this for up-front training cost and capacity limits.
MemTrapBench shows that all five tested memory frameworks underperform a no-memory baseline, with drops exceeding 10%, despite memories being accurately stored and task-relevant. This reveals a failure mode at the point of use that standard memory benchmarks miss.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Know It, Act on It: Investigating Memory Utilization in LLM Personalization
- MemTrapBench: Benchmarking Cognitive Traps in LLM Memory Use
- Memory is Reconstructed, Not Retrieved: Graph Memory for LLM Agents
- A Human-Inspired Reading Agent with Gist Memory of Very Long Contexts
- MeMo: Memory as a Model
- Understanding Users' Privacy Perceptions Towards LLM's RAG-based Memory
- Relational Gains, Privacy Strains: Exploring Users' Perceptions and Experiences with ChatGPT's Memory Feature
- Context Embeddings for Efficient Answer Generation in RAG