Can you just feed an AI all your documents at once, instead of building a search system on top?

This explores whether stuffing everything into a long context window can do the job of retrieval-augmented generation (RAG) — pulling in outside documents on demand — and the honest answer from the corpus is: partly, for one kind of task, and not for others. The cleanest evidence comes from the LOFT benchmark, where long-context models match RAG on *semantic* retrieval — finding the passage that means what you asked — with no special training. But they fall apart on structured, relational queries that require joining information across tables. Context length alone can't bridge that gap Can long-context LLMs replace retrieval-augmented generation systems?. So the replacement story is real, but narrow: it's a substitution for the fuzzy-matching half of retrieval, not the precise-lookup half.

The deeper problem is that a big context window isn't the same as a *usable* one. Reasoning accuracy drops from 92% to 68% with just 3,000 tokens of padding — far below the model's actual limit — and the decay is task-agnostic and survives chain-of-thought prompting Does reasoning ability actually degrade with longer inputs?. Worse, models often ignore what's sitting right in front of them when their training-time associations are strong enough to override the supplied text, so the answer reflects memorized priors rather than the document you handed over Why do language models ignore information in their context?. This is exactly the failure RAG was invented to avoid, and it doesn't disappear just because the context got longer.

There's also a hidden cost story. One line of work argues the long-context bottleneck isn't memory capacity at all — it's the *compute* needed to consolidate all that evicted context into the model's internal state, a process that scales like test-time reasoning and rewards multiple passes Is long-context bottleneck really about memory or compute?. That reframes "just use a longer context" as "pay more compute per query," which is precisely the efficiency argument RAG makes in the other direction: retrieve a little, reason over a lot less.

What's interesting is that the corpus doesn't frame this as long-context *vs.* RAG so much as a field splintering into hybrids. Human-inspired reading agents compress documents into "gist memories" before they even know the question, then fetch details only when needed — extending effective context 3–20× and beating retrieval baselines on long-document QA Can LLMs read long documents like humans do?. Compressive-memory systems fold retrieval, summarization, and response into a single model for conversation history, dropping the vector database entirely — though they follow a fragile inverted-U where too much reprocessing degrades below having no memory at all Can a single model replace retrieval for long-term conversation memory?. And in the opposite direction, agents that *grep* raw text with shell commands beat dense embeddings on entity-constrained multi-hop queries, recovering the lexical precision that embeddings blur Can direct corpus search beat embedding-based retrieval?.

The thing you didn't know you wanted to know: the RAG world is itself moving past the static-index picture that long-context is supposedly replacing. Retrieval research now argues that retrieval must adapt dynamically and interleave tightly with reasoning rather than run as a fixed first step How should systems retrieve and reason with external knowledge?, and some systems even grow their own corpus by writing verified generated answers back into the retrieval store Can RAG systems safely learn from their own generated answers?. So the honest framing isn't "will long context kill RAG" — it's that both are converging on the same design: retrieve less, reason more, and decide *when* to look things up rather than dumping everything in at once.