Can stateless multi-step retrieval capture evidence integration as well as dynamic memory?
This explores whether retrieval that treats each step independently — fetch, reason, repeat, with no carried-over state — can pull scattered evidence together as well as systems that keep a running, updatable memory across the search.
This explores whether stateless multi-step retrieval (each hop reasons fresh, then discards) can integrate evidence as well as systems that maintain a persistent, updatable memory. The corpus has a direct answer to this exact contest, and then a more interesting set of complications around it. The cleanest head-to-head is ComoRAG Can reasoning systems maintain memory across retrieval cycles?, which finds that a persistent memory workspace beats stateless multi-step retrieval by up to 11% on complex narrative questions — and the *reason* matters more than the number. Memory wins specifically where evidence has to be reconciled: the workspace lets the system notice contradictions between what it fetched on hop three and hop seven, then go back and resolve them. Stateless retrieval can't do that, because each step has already forgotten the others. So for genuine *integration* — not just gathering, but cross-checking — state seems to earn its keep.
But the corpus immediately pushes back on treating 'more memory' as the obvious answer. The memoryless camp argues that history is often baggage. Atom of Thoughts Can reasoning systems forget history without losing coherence? deliberately makes each reasoning state depend only on the current sub-problem, contracting the problem into a clean DAG so prior steps can't bloat or pollute the current one — and it preserves answer correctness while doing so. The lesson: accumulated history doesn't help if it accumulates noise. And memory can actively backfire — COMEDY Can a single model replace retrieval for long-term conversation memory? folds everything into one continuously-reprocessed memory blob and follows an inverted-U: past a point it degrades *below* having no memory at all, through misgrouping and context loss. So the real axis isn't stateless-vs-stateful; it's whether the state you keep is structured for reconciliation or just piling up.
That reframing is where the multi-step-retrieval side gets its strongest rebuttal. CoRAG Can retrieval be extended into multi-step chains like reasoning? shows stateless-ish retrieval chains can be made powerful by scaling them — longer chains, tree search at test time — turning retrieval into a tunable compute dial like reasoning tokens. And the integration quality may depend less on whether you hold state than on *what you fetch and how you decide*. METEORA Can rationale-driven selection beat similarity re-ranking for evidence? gets 33% better accuracy with half the chunks by selecting evidence via LLM-written rationales instead of raw similarity — better evidence per step lowers the burden on any integration layer. DeepRAG When should language models retrieve external knowledge versus use internal knowledge? frames each step as a decision of whether to retrieve at all, cutting noise from needless fetches; and uncertainty estimation Can simple uncertainty estimates beat complex adaptive retrieval? shows the model's own self-knowledge often beats elaborate adaptive machinery at deciding when to reach out. A disciplined stateless chain that fetches less but better can close much of the gap.
Where the gap stays stubborn is *global* reasoning — questions whose answer lives in the relationships between far-apart pieces, not in any one chunk. Here both flat stateless retrieval and naive memory struggle, and the corpus's answer is structure rather than state per se. Hierarchical research architectures Do hierarchical retrieval architectures outperform flat ones on complex queries? separate planning from synthesis to win on multi-hop queries, and MegaRAG Can multimodal knowledge graphs answer questions that flat retrieval cannot? builds a knowledge graph so cross-chapter questions become traversals instead of lucky retrievals. The unexpected takeaway: 'dynamic memory' is doing two different jobs we tend to conflate — holding a contradiction-resolution scratchpad (which stateless retrieval genuinely can't replicate) and providing a persistent *structure* of relationships (which a graph or hierarchy can provide without per-query state). Stateless multi-step retrieval can match dynamic memory on the second job if the structure is pre-built and the per-step decisions are sharp — but on the first, the live reconciliation of conflicting evidence, the persistent workspace still wins.
Sources 9 notes
ComoRAG demonstrates that iterative evidence acquisition with a persistent memory workspace outperforms stateless multi-step retrieval by detecting and resolving contradictions through deeper exploration, achieving up to 11% gains on complex queries.
Atom of Thoughts decomposes problems into DAGs and contracts them iteratively, ensuring each state depends only on the current problem—not prior steps. This memoryless approach eliminates historical baggage that bloats reasoning while maintaining answer equivalence.
COMEDY merges memory generation, compression, and response into one operation, tracking event recaps, user portraits, and relationship dynamics without vector-DB retrieval. However, empirical work shows continuous reprocessing follows an inverted-U curve, degrading below no-memory baseline due to misgrouping, context loss, and overfitting.
CoRAG extends chain-of-thought training to retrieval by using rejection sampling to generate intermediate retrieval chains. Test-time compute can scale through chain length and count, creating a compute dial—greedy decoding for speed or tree search for accuracy—just like reasoning-token scaling.
METEORA uses LLM-generated rationales with flagging instructions to select evidence, achieving 33% better accuracy with 50% fewer chunks than similarity re-ranking across legal, financial, and academic domains. The method also improves adversarial robustness substantially.
DeepRAG models each reasoning step as a Markov Decision Process where the model learns when to retrieve versus rely on parametric knowledge. The 21.99% improvement comes from better-targeted retrieval and elimination of noise from unnecessary external knowledge.
Calibrated token-probability uncertainty consistently beats multi-call adaptive retrieval on single-hop tasks and matches performance on multi-hop, using a fraction of the LM and retriever calls. The model's self-knowledge proves more reliable than external heuristics for deciding when to retrieve.
Separating query planning from answer synthesis into distinct components reduces interference and improves multi-hop query performance. This architectural principle mirrors documented benefits of separating planning from execution in agent design.
MegaRAG builds hierarchical multimodal knowledge graphs from text and visuals to answer cross-chapter, global questions that flat chunk retrieval cannot reach. The hierarchy supports abstraction levels from high-level summaries to page-specific details while treating images as first-class graph nodes.