Instead of searching a giant database every time an AI answers, could a tiny model just memorize what the search would find?
Can retrieval behavior be compressed into a small parametric decoder?
This explores whether the knowledge a retrieval system fetches from a big datastore at query time can be baked into a small trained model, so the search step disappears.
This explores whether the knowledge a retrieval system fetches from a big datastore at query time can be baked into a small trained model, so the search step disappears. The corpus says yes for one kind of retrieval and hints at where the limits sit.
The yes is Memory Decoder. It trains a small transformer to imitate what kNN-LM retrieval would have produced: a probability distribution over the next word, built by looking up similar contexts in a datastore. That small model plugs into any LLM by blending output probabilities, so there is no datastore search at inference. It keeps long-tail factual knowledge, stays semantically coherent, and cuts perplexity by 6.17 points across domains Can retrieval knowledge compress into a tiny parametric model?. The interesting part is that retrieval becomes a portable module instead of an index you have to host next to every model.
The evidence is about perplexity, meaning how well the model predicts text. It does not show that a small decoder can hand back an exact passage or record. Other notes suggest why that gap matters, though none of them test Memory Decoder directly. Retrieval systems already hit limits from compression: embedding dimension caps which sets of documents a system can represent at all Where do retrieval systems fail and why?. A small verifier that reads full token-to-token similarity maps rejects structural near-misses that compressed vectors let through Can verification separate structural near-misses from topical matches?. State-space models, with their fixed-size internal state, provably lose to transformers at copying and retrieving from context Can state-space models match transformers at copying and retrieval?. A small decoder is also a fixed-size store, so exact lookup is where I'd expect it to strain, but the corpus doesn't settle that.
GrepSeek makes the opposite bet. It skips compression, and even embeddings: an agent issues shell commands over raw text and does better on entity-constrained multi-hop questions, because plain text matching keeps precision where embeddings blur similar entities together Can direct corpus search beat embedding-based retrieval?. Put side by side, the two approaches split the job. Compress retrieval into weights for cheap, fluent, long-tail knowledge. Stay literal when the answer has to be exact.
Compression also isn't automatically safe. COMEDY replaces retrieval with a single model that keeps rewriting running summaries of a conversation. That is a different mechanism from a trained decoder, but it shows the same risk: repeated reprocessing follows an inverted-U curve and eventually falls below having no memory at all, through misgrouping, lost context and overfitting Can a single model replace retrieval for long-term conversation memory?. A separate question is when to retrieve at all. Calibrated token-probability uncertainty beats multi-call adaptive retrieval at a fraction of the cost Can simple uncertainty estimates beat complex adaptive retrieval?. The corpus doesn't say whether a compressed decoder could use that signal to decide how much to trust itself.
Sources 7 notes
Memory Decoder successfully compresses kNN-LM retrieval distributions into a small transformer that plugs into any LLM via output interpolation. It preserves long-tail factual knowledge while maintaining semantic coherence, reducing perplexity by 6.17 points across domains.
RAG systems fail at three structural levels: adaptive triggering (fixed intervals waste context), semantic-task mismatch (embeddings measure association, not relevance), and mathematical limits (embedding dimension constrains representable document sets). These require fundamentally different retrieval approaches, not tuning.
A two-stage pipeline—pooled-cosine recall followed by a small Transformer verifier operating on token-token similarity maps—reliably rejects structural near-misses that MaxSim-style late interaction cannot. The verifier succeeds because it operates on full token interaction patterns rather than compressed vectors.
Two-layer transformers can copy exponentially long strings while state-space models are fundamentally limited by their fixed-size latent state. Empirically, transformers dramatically outperform SSMs at copying and context retrieval in both synthetic and pretrained settings.
GrepSeek trains agents to retrieve via executable shell commands over raw text, achieving better multi-hop performance on entity-constrained queries than dense embeddings. The approach scaffolds unstable search mechanics with supervised trajectories, then refines task-oriented behavior through reinforcement learning.
Show all 7 sources
COMEDY merges memory generation, compression, and response into one operation, tracking event recaps, user portraits, and relationship dynamics without vector-DB retrieval. However, empirical work shows continuous reprocessing follows an inverted-U curve, degrading below no-memory baseline due to misgrouping, context loss, and overfitting.
Calibrated token-probability uncertainty consistently beats multi-call adaptive retrieval on single-hop tasks and matches performance on multi-hop, using a fraction of the LM and retriever calls. The model's self-knowledge proves more reliable than external heuristics for deciding when to retrieve.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Chain-of-Retrieval Augmented Generation
- Towards Agentic RAG with Deep Reasoning: A Survey of RAG-Reasoning Systems in LLMs
- CLaRa: Bridging Retrieval and Generation with Continuous Latent Reasoning
- A Human-Inspired Reading Agent with Gist Memory of Very Long Contexts
- A New Role for Relevance: Guiding Corpus Interaction in Agentic Search
- BM25 Wins at Scale: A Scaling Study of Retrieval-Augmented Generation Paradigms
- Training for Compositional Sensitivity Reduces Dense Retrieval Generalization
- UR2: Unify RAG and Reasoning through Reinforcement Learning