Why do cross-product features memorize better than dense embeddings?
This explores why sparse cross-product features (exact memorized combinations) hold onto specific item-pairings that dense embeddings blur away — and what that trade-off reveals about how models store knowledge.
This is really about a division of labor between two ways of storing knowledge: cross-product features memorize exact, observed combinations, while dense embeddings generalize by mapping things into a smooth space where similar items sit close together. The Wide & Deep work makes the case directly — the 'wide' cross-product tower exists precisely to capture the rare, specific pairings that the 'deep' embedding tower smooths over. Trained jointly, each half specializes: the deep side handles common, generalizable cases, so the wide side can stay small and just patch the exceptions the embeddings can't represent without overfitting Can one model memorize and generalize better than two? Can one model handle both memorization and generalization?.
The deeper reason embeddings struggle to memorize is what they actually measure. Dense vectors encode *semantic association* — co-occurrence and similarity — not exact relevance. That makes them great at 'these two things are related' and bad at 'this specific user picked this specific rare item.' In production, underspecified inputs surface many wrong-but-associated candidates, because the embedding can't distinguish a precise match from a merely-similar one Do vector embeddings actually measure task relevance?. A cross-product feature has no such fuzziness: it either fired for that exact combination or it didn't. Memorization is, in a sense, the refusal to interpolate.
There's a complementary clue in how networks allocate representation at all. Models learn *dense* activations for familiar training data and fall back to *sparse* ones for the unfamiliar Is representational sparsity learned or intrinsic to neural networks?. Rare combinations are, almost by definition, the unfamiliar tail — exactly the region where a smooth embedding has little signal and a sparse, explicit feature earns its keep.
What's striking is that this isn't a quirk of recommenders — it's a recurring theme that *structural bias often beats raw capacity.* ESLER, a single-layer linear model with a constraint forbidding items from predicting themselves, beats most deep collaborative filtering: forcing prediction through explicit item relationships matters more than model depth Can a linear model beat deep collaborative filtering?. And VQ-Rec deliberately *discretizes* text into codes rather than trusting continuous embeddings, because the discrete intermediate breaks text-similarity bias and lets the model store domain-specific lookups embeddings would otherwise wash out Can discretizing text embeddings improve recommendation transfer? Can discrete codes transfer better than text embeddings?.
The thing you might not have expected: 'memorize vs. generalize' isn't a flaw to fix but an architecture to design around. The reason the best systems keep both a sparse and a dense path — Wide & Deep, or even Titans separating compressed long-term memory from attention Can neural memory modules scale language models beyond attention limits? — is that smoothing and exact recall are genuinely different jobs, and a single representation can't be optimal at both at once.
Sources 8 notes
Wide & Deep models train memorization (cross-product features) and generalization (embeddings) together, allowing each component to specialize: the wide part becomes small because deep handles common cases, and deep doesn't overfit rare items because wide captures them. Ensembling requires both halves full-size.
Wide & Deep architectures train a sparse cross-product tower and a dense embedding tower together, allowing the wide part to patch only the deep part's weaknesses. This joint approach requires smaller models than ensemble methods.
Embeddings encode co-occurrence patterns, making semantically close but role-distinct concepts highly similar. This works in simple demos but fails in production where underspecified queries have many wrong-but-associated candidates.
During pretraining, neural networks develop dense activations for familiar training data and default to sparse representations for unfamiliar inputs. This trend emerges without task-specific fine-tuning and reflects how models consolidate knowledge through exposure.
ESLER, a single-layer linear autoencoder constrained so items cannot predict themselves, outperforms most deep CF models. The constraint forces prediction through item relationships, and negative weights encoding anti-affinity prove essential—structural bias matters more than model capacity.
VQ-Rec uses product quantization to map item text to discrete codes that index learned embeddings, breaking the tight coupling between text and recommendations. This decoupling prevents text-similarity bias and allows lookup tables to adapt to new domains without retraining the text encoder.
VQ-Rec demonstrates that mapping item text to discrete codes via product quantization, then to embeddings, improves cross-domain transfer compared to direct text encoding. The discrete intermediate reduces text bias and enables efficient per-domain fine-tuning.
Titans architecture separates attention (short-term, quadratic) from neural memory (long-term, compressed), prioritizing surprising tokens for storage. The model outperforms standard Transformers and linear RNNs across tasks while scaling to 2M+ token contexts without quadratic penalties.