The cost gap between AI search and AI chat isn't about model size — it's about which direction the model reads.

This explores why encoder models are the cheaper workhorse for turning document corpora into searchable representations, and the corpus points to one root cause: it's the attention mechanism, not the model size. An encoder reads a document with *bidirectional* attention — every token can see every other token at once — so it builds a single dense representation of the whole passage in one forward pass. A decoder-only model uses *causal masking*: each token can only look leftward, which is exactly what you want for generating text word-by-word but throws away information when the job is to summarize a passage into one vector. The clearest evidence that masking (not scale) is the bottleneck comes from work showing that if you switch a decoder to bidirectional attention and retrain slightly, it suddenly becomes a competitive encoder Why do decoder-only models underperform as text encoders?.

The efficiency win is also empirical and current, not a legacy artifact. A modernized encoder trained on trillions of tokens matches or beats much larger decoders on retrieval quality while running roughly twice as fast and using less memory Can encoder models compete with large language models for retrieval?. The reason compounds at corpus scale: encoders pre-compute one embedding per document *once*, offline, and then retrieval is just fast vector comparison. A decoder-based approach that stuffs documents into a long context window instead pays a compute cost every single query, and — importantly — the bottleneck there isn't running out of memory but the compute needed to transform all that raw context into usable internal state Is long-context bottleneck really about memory or compute?.

What's genuinely interesting is that efficiency isn't the encoder's only edge — the trade is task-shaped. Decoder-based long-context models can actually *replace* retrieval for open-ended semantic questions, reading a corpus directly and answering, but they fall apart on structured queries that need joins across tables or exact relational logic Can long-context LLMs replace retrieval-augmented generation systems?. So the encoder's cheap-and-precise profile matters most exactly where decoders are weakest. And there's a further twist: for entity-heavy, lexically precise lookups, even dense encoder embeddings blur distinct entities together, and an agent issuing literal text searches over raw files can beat embedding retrieval outright Can direct corpus search beat embedding-based retrieval?.

The deeper architectural theme running underneath is that transformers are simply better at addressing and pulling specific items out of a context than architectures with a fixed-size compressed state — provably so against state-space models on copying and retrieval Can state-space models match transformers at copying and retrieval?. Encoders inherit that retrieval strength while shedding the generative overhead. The takeaway you might not have expected: the field's move toward decoder-only everything isn't a one-way ratchet. For the specific job of indexing a corpus, the bidirectional encoder remains the efficient choice not because it's smaller but because seeing the whole passage at once is the right shape for the task.