Turns out you can teach old-school search models new tricks by borrowing the training playbook from ChatGPT-era AI.

This explores which training-side advances that were developed for decoder LLMs have been carried back into encoder-only models like BERT. The corpus has one note that answers this head-on, and it's a good one: ModernBERT is essentially an exercise in porting the decoder era's engineering back into an encoder Can encoder models compete with large language models for retrieval?. The concrete transfers are a much larger pretraining budget (2 trillion tokens, on par with modern LLMs rather than the few-hundred-billion of original BERT), a long 8192-token context window borrowed straight from decoder scaling work, and the efficiency plumbing that made big decoders trainable. The payoff is that a modernized encoder beats larger decoder models on retrieval and classification while running roughly twice as fast and using less memory — so the migration to decoders isn't the one-way street it's often assumed to be.

What's worth noticing is that most of what transfers is *scale and data discipline*, not architecture. That reframes the question: the interesting LLM-era optimizations aren't decoder-specific tricks, they're general laws about how to spend training compute. The clearest example in the corpus is the finding that optimal language-model learning can be derived from a lossless-compression objective, yielding a 'Learning Law' where every example contributes equally in the optimal process Does optimal language model learning maximize data compression?. That result improves the *coefficients* of scaling laws, not just the constants — and nothing about it is tied to a decoder. A compression-maximizing data curriculum is exactly the kind of thing that should transfer to an encoder's masked-language pretraining as cleanly as it does to next-token prediction.

The corpus also gives a useful caution about what you'd want to be careful *not* to import. RL post-training, a signature LLM optimization, has a hidden cost: it collapses format diversity, amplifying a single dominant pretraining distribution within the first epoch while suppressing the alternatives Does RL training collapse format diversity in pretrained models?. For an encoder whose whole job is producing rich, discriminative representations for retrieval, that kind of diversity collapse is a liability, not a feature — a reminder that 'transfers back' should mean the pretraining and efficiency advances more than the alignment-style post-training.

So the sharp version of the answer: the optimizations that flow back to encoders are the data-and-compute ones — trillion-token corpora, long-context training, and compression-optimal learning schedules — plus the systems efficiency work that made those budgets affordable. The reader-surprising bit is that this traffic runs both directions. Architectural progress isn't a monotonic drift toward decoders; when you feed an old encoder recipe the new training budget, it reclaims the retrieval and classification niche outright Can encoder models compete with large language models for retrieval?.