Why do retrieval systems fail on queries that never mention needed facts?
Memory retrieval assumes needed information will resemble the query text, but world knowledge often connects facts indirectly. This explores when that assumption breaks and what happens when it does.
The paper argues that retrieval-based memory rests on an assumption "so natural that it is rarely stated": a memory that is needed will resemble the query that needs it. World knowledge breaks that assumption. A tree-nut allergy should change the answer to a macaron request through the almond-flour ingredient, yet "the two texts share no cue a retriever can see." The authors call this the implicit-association blind spot and build InMind, a 125-task, expert-verified benchmark across ten life domains, to measure it. The headline numbers are stark: with the decisive memory placed in context, the backbone answers 84.0% of indirect queries, while six vector, graph, and agentic memory systems reach at most 14.4% when the same memory must be retrieved.
The mechanism the paper gives is structural rather than a matter of forgetting. Its introduction notes that an evaluated system recalled the allergy on demand seconds before recommending almond flour, and the abstract reports the systems recall the same facts on demand at up to 100%. The memory was stored and the model can reason from it, but it "was never brought to bear at the one moment it mattered." The paired controls are what license this reading, because they separate three explanations that existing evaluations conflate: the fact was never stored, the model lacks the bridging knowledge, or the fact was stored and never surfaced. The 84.0% in-context result rules out the second, and the on-demand recall rules out the first, which leaves surfacing. The one intervention the excerpt reports points the same way: an embedding with eight times the dimensionality raises "answer-blind target recall for every system yet leaves the gap essentially intact."
This locates a failure upstream of the one in Why do LLM agents remember preferences but not act on them?. That paper finds agents recalling a preference and then not using it, with comprehension errors dominating; here the model does use the memory when handed it, and the loss happens before the model sees anything. The two are compatible as different stages, but the excerpts do not reconcile them. It also gives a concrete failure mode for the decomposition argued in How should we actually evaluate agent memory systems?, where a recall-on-demand score would have passed these systems. And it qualifies Does retrieved memory quality depend on its functional role?: when relevance is judged by resemblance to the query, a memory can be essential and still look irrelevant.
The excerpt is short on what would make this actionable. It does not name the six systems or the backbone, say how indirect queries were built or scored beyond being expert-verified, break results out by domain, or define answer-blind target recall. It carries no discussion passage, so no remedy is proposed. What it supports is narrower and still useful: on this benchmark, recall on demand is not evidence that a memory will surface when a user's request never mentions it, and a larger embedding alone did not close the gap for the systems tested.
Inquiring lines that read this note 3
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
Why does memory consolidation cause performance regression in continual learning? How should retrieval systems handle complex multi-step reasoning?Related concepts in this collection 3
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
-
Why do LLM agents remember preferences but not act on them?
Research using paired recall and behavioral tests explores why language models often retrieve user preferences correctly yet fail to apply them in actual responses, especially for health-related requests.
a later-stage failure: there recalled preferences go unused; here the memory never surfaces for indirect queries
-
How should we actually evaluate agent memory systems?
Current benchmarks score agent memory by task success alone, hiding critical design questions about cost, trade-offs, and robustness. What would evaluation reveal if we decomposed memory into its core data-management stages?
paired controls localize this failure to retrieval, which the decomposition argument calls for
-
Does retrieved memory quality depend on its functional role?
Conversational RAG systems retrieve context to improve responses, but does the *type* of memory matter as much as its relevance score? This explores whether different memory roles (clarifying vs. irrelevant) drive response quality differently.
relevance by resemblance misses memories that matter through world knowledge
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- On the Theoretical Limitations of Embedding-Based Retrieval
- Towards Agentic RAG with Deep Reasoning: A Survey of RAG-Reasoning Systems in LLMs
- Keep It InMind: Benchmarking the Implicit-Association Blind Spot in Agent Memory
- ComoRAG: A Cognitive-Inspired Memory-Organized RAG for Stateful Long Narrative Reasoning
- Think-in-Memory: Recalling and Post-thinking Enable LLMs with Long-Term Memory
- Memory Makes the Difference: Evaluating How Different Memory Roles Shape Conversational Agents
- MeMo: Memory as a Model
- Continual Learning Mechanisms Compose for Long-Horizon Memorization
Original note title
retrieval-based agent memory assumes a needed memory will resemble the query — the implicit-association blind spot breaks it on indirect queries