Should an AI look things up first and then think, or keep checking sources as it reasons?
How should retrieval and reasoning be integrated architecturally?
This explores how to wire retrieval and reasoning together at the design level: whether retrieval should be a step before thinking or something woven through it, and what the corpus says about the shape of that weaving.
This explores how to wire retrieval and reasoning together at the design level: whether retrieval is a step that happens before thinking, or something interleaved with it. The corpus leans strongly toward interleaving. The core finding is that fixed pipelines, where you fetch documents and then prompt the model, fail because when and what to retrieve depends on what the reasoning needs at that moment. Systems have to rag|couple retrieval with inference rather than just prepending it. The most concrete version treats each reasoning step as a decision point. retrieval-augmented-reasoning-as-markov-decision-process-enables-per-step-parame|DeepRAG frames this as a Markov Decision Process in which the model learns per step whether to retrieve or trust what it already knows. That gained about 22% accuracy, partly because skipping unnecessary retrieval removes noise. How should retrieval and reasoning integrate in RAG systems? adds that step-level feedback, not just a final-answer score, is what makes this tight coupling trainable.
A second design question is what the reasoning loop does with what it retrieves. Retrieve-then-reason treats memory as a lookup table. moving-relational-reasoning-from-storage-into-retrieval-lets-agent-memory-be-rec|MRAgent instead reconstructs memory by walking a graph. It prunes paths as evidence accumulates, which gave gains of up to 23% and lower token cost than fixed pipelines. Statefulness matters too. stateful-narrative-reasoning-requires-iterative-evidence-acquisition-and-knowled|ComoRAG keeps a persistent memory workspace across retrieval rounds, so it can notice contradictions between what it found earlier and later and dig deeper. Stateless multi-step retrieval can't do that. The shape of the memory matters as well. hypergraph-memory-lets-multi-step-rag-combine-facts-over-time-pairwise-edges-can|Hypergraph memory lets three or more entities bind into one relation, so a joint constraint isn't lost when it's broken into pairwise links.
A different axis is how you divide the labor. hierarchical-research-architectures-that-separate-query-planning-from-answer-syn|Hierarchical designs that separate query planning from answer synthesis beat flat ones on multi-hop questions because the two jobs stop interfering with each other. How should reasoning systems actually be architected? makes the same point at the level of reasoning itself: RL post-training mostly teaches a model when to activate mechanisms it already has. That suggests separating the timing decision from the execution machinery. Interleaving and separation can sound like opposites, but they aren't. Retrieval is woven into the loop, while planning and synthesis sit in distinct components. Routing is a third version of the same idea. cognitive-fit-theory-applied-to-rag-routing-queries-to-task-appropriate-knowledg|StructRAG trains a router to pick the knowledge structure that fits the question (table, graph, catalogue, plain chunks) instead of forcing one format on everything.
The corpus also explains why the architecture has to change and can't just be tuned. Where do retrieval systems fail and why? argues the failures are structural: fixed-interval triggering wastes context, embeddings measure association rather than relevance, and embedding dimension caps which document sets can be represented at all. Two neighboring notes show the limits from either side. long-context-llms-can-subsume-standard-rag-for-semantic-retrieval-but-fail-on-co|Long-context models can replace RAG for semantic lookup but still can't do relational joins over structured data, so more context is not a substitute for a better structure. And identity-sensitive-matching-should-be-a-distinct-verification-task-downstream-of|a small learned verifier downstream of cheap recall catches near-misses that vector similarity can't tell apart from true matches. That is another case of splitting a job into stages instead of asking one embedding to do everything.
The pattern across these notes is that retrieval works best as an ongoing conversation with the reasoning process: decide when to look, look in a structure that fits the question, keep state between looks, and verify what comes back. What the corpus doesn't settle is how much of this needs training and how much can be done through scaffolding. The strongest results (DeepRAG, MRAgent) involve learned policies, but the notes don't compare them head to head with well-designed fixed pipelines.
Sources 12 notes
Research shows that tight coupling between retrieval and reasoning—via Markov Decision Processes and step-level feedback—substantially improves accuracy and efficiency. Graph-based retrieval and metacognitive monitoring address limitations of vector embeddings and prevent retrieval failures on compositional tasks.
Separating query planning from answer synthesis into distinct components reduces interference and improves multi-hop query performance. This architectural principle mirrors documented benefits of separating planning from execution in agent design.
StructRAG demonstrates that selecting knowledge structure type based on query demands—via DPO-trained router choosing among tables, graphs, algorithms, catalogues, and chunks—improves knowledge-intensive reasoning over standard retrieval. The approach grounds this in cognitive load and cognitive fit theory from cognitive science.
RAG systems fail at three structural levels: adaptive triggering (fixed intervals waste context), semantic-task mismatch (embeddings measure association, not relevance), and mathematical limits (embedding dimension constrains representable document sets). These require fundamentally different retrieval approaches, not tuning.
ComoRAG demonstrates that iterative evidence acquisition with a persistent memory workspace outperforms stateless multi-step retrieval by detecting and resolving contradictions through deeper exploration, achieving up to 11% gains on complex queries.
Show all 12 sources
MRAgent achieves up to 23% gains on reasoning tasks by reconstructing memory through active graph traversal that prunes paths based on accumulated evidence, while reducing token and runtime cost compared to fixed-retrieval pipelines.
Research shows fixed retrieval pipelines fail because retrieval timing and content must adapt to reasoning needs. Systems must couple retrieval with inference, not just prepend it.
Research shows RL post-training teaches models *when* to use reasoning mechanisms that pre-training already provides. Decoupled architectures, latent reasoning in continuous space, and interleaved action-grounding all outperform monolithic chain-of-thought approaches.
HGMem organizes retrieved evidence as hyperedges rather than flat lists or binary graphs, allowing three or more entities to bind into single relations without decomposition. This structure accumulates coherent knowledge across retrieval steps, trading representational complexity for constraint expressiveness.
The LOFT benchmark shows LCLMs match RAG on semantic retrieval without explicit training, but cannot execute relational queries requiring joins across structured tables. Context length alone cannot bridge this gap.
DeepRAG models each reasoning step as a Markov Decision Process where the model learns when to retrieve versus rely on parametric knowledge. The 21.99% improvement comes from better-targeted retrieval and elimination of noise from unnecessary external knowledge.
A two-stage pipeline—pooled-cosine recall followed by a small Transformer verifier operating on token-token similarity maps—reliably rejects structural near-misses that MaxSim-style late interaction cannot. The verifier succeeds because it operates on full token interaction patterns rather than compressed vectors.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Towards Agentic RAG with Deep Reasoning: A Survey of RAG-Reasoning Systems in LLMs
- Chain-of-Retrieval Augmented Generation
- You Don't Need Pre-built Graphs for RAG: Retrieval Augmented Generation with Adaptive Reasoning Structures
- UR2: Unify RAG and Reasoning through Reinforcement Learning
- DeepRAG: Thinking to Retrieval Step by Step for Large Language Models
- Fact, Fetch, and Reason: A Unified Evaluation of Retrieval-Augmented Generation
- RAG-R1 : Incentivize the Search and Reasoning Capabilities of LLMs through Multi-query Parallelism
- Deep Research: A Systematic Survey