Can encoder-only architectures match decoder-based sequential models for recommendation?
This explores whether bidirectional encoder-only transformers (which see a whole sequence at once) can hold their own against the autoregressive, decoder-style models that predict the next item one step at a time — the dominant paradigm in sequential recommendation.
This explores whether bidirectional encoder-only transformers can match the autoregressive, next-item-prediction models that dominate sequential recommendation — and the corpus says yes, with a clever twist. The most direct evidence is Sequential Masked Modeling, which adapts encoder-only transformers to session-based recommendation using "penultimate-token masking" (masking the second-to-last item so the encoder learns to predict what comes next) plus sliding-window augmentation. The striking result isn't just that it matches other single-session methods — it rivals cross-session recommenders that have access to far richer user history, despite seeing only one session Can single sessions alone rival history-rich recommendation?. The architectural choice (encoder vs. decoder) turns out to matter less than how you frame the prediction task.
That lesson — structure and framing beat raw architectural muscle — echoes loudly across the rest of the corpus. The most pointed example is the EASE/ESLER line of work, where a shallow *linear* item-item model with one trick (forcing the diagonal to zero so an item can't predict itself) beats deep neural autoencoders on most datasets Can simpler models beat deep networks for recommendation systems? Can a linear model beat deep collaborative filtering?. In both cases the win comes from a structural prior — a constraint that forces the model to learn relationships through other items — rather than from more parameters or a fancier sequence model. Encoder-only recommendation succeeds for the same reason: the masking objective is a structural prior that makes a non-autoregressive model behave as if it understood sequence.
There's a second framing worth pulling in: you don't have to pick "encoder vs. decoder" at all. P5 recasts recommendation as text-to-text and trains a single encoder-decoder across five task families, matching task-specific models and transferring zero-shot to new items Can one text encoder unify all recommendation tasks?. So the encoder/decoder distinction partly dissolves once you treat recommendation as language modeling — the interesting question becomes what representation the model consumes, not which half of the transformer you keep.
That shifts attention to representation choices that cut across architecture entirely. VQ-Rec maps item text to discrete codes before embedding, which decouples the recommender from any particular text encoder and transfers across domains better than raw text embeddings Can discretizing text embeddings improve recommendation transfer? Can discrete codes transfer better than text embeddings?. Meanwhile AMP-CF shows that *where* you spend modeling capacity — splitting a user into multiple attention-weighted personas conditioned on the candidate item — buys both accuracy and built-in explanations Can attention mechanisms reveal which user taste explains each recommendation? Can modeling multiple user personas improve recommendation accuracy?.
The thing you didn't know you wanted to know: the field keeps rediscovering that the recommendation "backbone" is surprisingly fungible. Encoder-only can match decoder-only, a linear model can match a deep one, and a text-to-text model can absorb both — because in recommendation the leverage lives in the prior you bake in (a masking objective, a zero-diagonal constraint, candidate-conditioned personas) far more than in the sequence-modeling machinery you happen to choose.
Sources 8 notes
Sequential Masked Modeling adapts encoder-only transformers for session-based recommendation using penultimate-token masking and sliding-window augmentation. Across three datasets, this single-session approach consistently outperforms other single-session methods and rivals cross-session approaches with richer user history.
EASE, a shallow linear item-item weight matrix with diagonal constrained to zero, beats deep neural baselines on most datasets. The constraint forces generalization by forbidding self-prediction, while learned negative weights capture item dissimilarity—a structural prior more valuable than model capacity.
ESLER, a single-layer linear autoencoder constrained so items cannot predict themselves, outperforms most deep CF models. The constraint forces prediction through item relationships, and negative weights encoding anti-affinity prove essential—structural bias matters more than model capacity.
P5 converts user-item interactions and metadata into natural language and trains a single encoder-decoder across five recommendation task families, matching task-specific models while achieving zero-shot transfer to new items and domains. Unification trades efficiency for composability.
VQ-Rec uses product quantization to map item text to discrete codes that index learned embeddings, breaking the tight coupling between text and recommendations. This decoupling prevents text-similarity bias and allows lookup tables to adapt to new domains without retraining the text encoder.
VQ-Rec demonstrates that mapping item text to discrete codes via product quantization, then to embeddings, improves cross-domain transfer compared to direct text encoding. The discrete intermediate reduces text bias and enables efficient per-domain fine-tuning.
AMP-CF represents each user as multiple latent personas weighted dynamically by candidate item. This makes recommendations both diverse and interpretable—each suggestion traces to the specific persona preference it satisfies—without requiring post-hoc reranking.
AMP-CF separates user representation into latent personas weighted by attention to the candidate item. This candidate-conditional approach improves accuracy by adapting the user representation at prediction time and produces inherent explanations for why items were recommended.