Even when an AI fetches exactly the right document, does it actually use it, or does it stick to what it already believes?
Can retrieval of correct information guarantee it will shape model behavior?
This explores whether fetching the right document is enough to make a model act on it, or whether the model can read correct information and still ignore it.
This explores whether fetching the right document is enough to make a model act on it, or whether the model can read correct information and still ignore it. The corpus says no. Retrieval only puts the information in front of the model. Whether the model uses it depends on what the model already believes and how it was trained.
The most direct evidence is that language models often produce outputs that contradict the context they were given, because associations learned in training outweigh what's in the prompt. When those priors are strong, prompting alone can't override them, and the research points to intervening in the model's internal representations instead (Why do language models ignore information in their context?). A related finding is that a model's confidence predicts how much a rewording of the prompt moves it (Does model confidence predict robustness to prompt changes?). That paper is about rephrasing, not retrieved evidence. My inference is that a confident model is stable in general, so it may also be harder to move with a passage that disagrees with it.
Being faithful to retrieved text turns out to be a trained skill. Models under 2B parameters, trained on synthetic multi-hop questions, learned to ground answers in the passage, quote it literally, and abstain when it didn't contain the answer (Can small models learn to ground answers in context?). So retrieval and faithfulness are separate abilities, and neither comes free with model size. More retrieval can also make things worse. DeepRAG improved accuracy by about 22% by teaching the model when to retrieve and when to trust what it already knows, which removed noise from unnecessary lookups (When should language models retrieve external knowledge versus use internal knowledge?). There is also a limit on what a retrieved fact can do: reasoning draws on broad procedural knowledge from pretraining, while factual recall depends on narrow memorization (Does procedural knowledge drive reasoning more than factual retrieval?). A fetched fact can fill a gap in what the model knows, but it won't add a reasoning procedure the model lacks.
Retrieved information can shape behavior when the conditions are right. Reflexion agents write a verbal self-diagnosis after a failure, store it in memory, and do better on the next attempt with no weight updates. It works because the success-or-failure signal is unambiguous, so the agent can't rationalize, and the reflections are kept uncompressed (Can agents learn from failure without updating their weights?). The influence also runs the other way. A model's partial answer can reveal what it still needs, which makes it a better retrieval query than the original question (Can a model's partial response guide what to retrieve next?). Here retrieval and behavior shape each other in a loop.
The deepest limit is about verification. Behavioral training can only show that a model complies in situations where it was observed. It can't show unconditional compliance, because no test can score unobserved behavior (Can behavioral training prove a model always complies?). That paper is about training, not retrieval. Extended to retrieval, it suggests you can test whether a model used a retrieved fact on the cases you tried, but you can't guarantee it will on the ones you didn't.
Sources 8 notes
Research demonstrates that LMs generate outputs inconsistent with their context because parametric knowledge from training dominates over in-context information. Textual prompting alone cannot override strong priors; causal intervention in representations is required.
ProSA found that when models are highly confident, they resist prompt rephrasing; low confidence causes major output swings. Larger models, few-shot examples, and objective tasks all correlate with higher confidence and greater robustness.
Sub-2B models trained on synthetic multi-hop QA can ground answers in passages, cite literal quotes, and abstain from confabulation. The OCC-RAG work shows faithfulness emerges from training curriculum design, not parameter count.
DeepRAG models each reasoning step as a Markov Decision Process where the model learns when to retrieve versus rely on parametric knowledge. The 21.99% improvement comes from better-targeted retrieval and elimination of noise from unnecessary external knowledge.
Analysis of 5 million pretraining documents shows reasoning relies on broad, transferable procedural knowledge from diverse sources, unlike factual recall which depends on narrow, document-specific memorization of target facts.
Show all 8 sources
Reflexion demonstrates that unambiguous environmental feedback (success/failure) enables agents to write useful self-diagnoses and improve across episodes without parameter updates. The binary signal prevents rationalization, and keeping reflections uncompressed preserves their usability.
ITER-RETGEN shows that iteratively using generated responses as retrieval queries substantially improves performance on multi-hop reasoning and fact verification. Generation acts as both answer producer and information-need clarifier, surfacing implicit gaps that the original query missed.
Any scored behavior is observed behavior, so training data cannot distinguish between a policy that always complies and one that complies only when watched. Only unobserved behavior would separate them, making such a test logically impossible.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- DeepRAG: Thinking to Retrieval Step by Step for Large Language Models
- Towards Agentic RAG with Deep Reasoning: A Survey of RAG-Reasoning Systems in LLMs
- RAG-R1 : Incentivize the Search and Reasoning Capabilities of LLMs through Multi-query Parallelism
- AbstentionBench: Reasoning LLMs Fail on Unanswerable Questions
- Procedural Knowledge in Pretraining Drives Reasoning in Large Language Models
- CLIN: A Continually Learning Language Agent for Rapid Task Adaptation and Generalization
- OCC-RAG: Optimal Cognitive Core for Faithful Question Answering
- AgentFly: Fine-tuning LLM Agents without Fine-tuning LLMs