Can one model memorize and generalize better than two?
Does training memorization and generalization components jointly in a single model outperform training them separately and combining their predictions? This matters for building efficient recommendation systems that handle both rare and common user behaviors.
The recommender ranking problem demands two opposing capabilities. Memorization — learning frequent feature co-occurrences — fits cross-product transformations on sparse features (e.g. "user installed Netflix AND impression is Pandora"). Generalization — predicting on never-seen feature combinations — fits dense embeddings that map sparse features into continuous space. Cross-products fail to generalize across unseen pairs; dense embeddings over-generalize and produce nonzero predictions for niche items the user shouldn't see.
The Wide & Deep insight is that you can put both halves into one model and train them jointly rather than ensembling separately-trained models. The technical distinction matters: in an ensemble, each component is optimized standalone and combined at inference, which means each must be full-size to perform reasonably alone. In joint training, the wide part is optimized knowing the deep part exists, so it only needs to complement the deep model's weaknesses with a small number of cross-product features. Gradients from the output flow into both branches simultaneously via mini-batch SGD.
The result is a model where memorization and generalization are not two competing strategies forced into a weighted average but two specialized roles in one network. The wide part becomes small and surgical because the deep part handles the common cases; the deep part doesn't have to learn rare exceptions because the wide part captures them.
Inquiring lines that read this note 22
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
Why does consolidated memory sometimes hurt continual learning more than no memory? What are the trade-offs between memory capacity and continuous task learning?- How much does memorization capacity limit a model's ability to learn new information?
- How do the three grokking phases connect to memorization capacity limits?
- Can data pruning strategies exploit the finite nature of memorization capacity?
- How do layer-wise versus parameter-wise merging strategies affect information retention?
- How does distributional shift toward rare inputs change memorization reliance?
- Can the joint-training principle extend beyond memorization and generalization pairs?
- What distinguishes data that generalizes broadly from task-specific memorization?
- How do complementary learning systems explain the need for fast and slow consolidation?
- How can a forgetting policy preserve rare knowledge while preventing over-generalization?
- Why does semantic deduplication reduce memorization in fine-tuned models?
- When does training a memory model beat RAG or fine-tuning?
- Does composing multiple continual learning mechanisms reduce forgetting more than single approaches?
Related concepts in this collection 4
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
-
Can one model handle both memorization and generalization?
Recommenders face a tradeoff between memorizing seen patterns and generalizing to new ones. Can a single architecture satisfy both needs without the cost of ensemble methods?
extends: paired statement of the same Wide&Deep result emphasizing the parameter-efficiency angle
-
Can autoencoders solve the cold-start problem in recommendations?
Explores whether deep autoencoders combining collaborative filtering with side information can overcome the cold-start problem where new users or items lack rating history.
extends: same hybridization-via-joint-training principle generalizes beyond CF+CBF to memorization+generalization
-
How can user vectors capture diverse interests without exploding in size?
Fixed-length user vectors compress all interests into one representation, losing information about varied tastes. Can we represent diverse interests efficiently without expanding dimensionality?
complements: DIN extends Wide&Deep with attention against candidates — same industrial-architecture lineage
-
Why do ranking systems need to model selection bias explicitly?
Explores how training data from current rankers creates feedback loops that reinforce past decisions. Understanding this mechanism helps explain why naive approaches fail in production ranking systems.
complements: MMoE generalizes joint-training to multi-task multi-objective — same joint-vs-separate optimization lesson
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- Continual Learning Mechanisms Compose for Long-Horizon Memorization
- Wide & Deep Learning for Recommender Systems
- SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-training
- Scaling can lead to compositional generalization
- Why Larger Models Learn More: Effects of Capacity, Interference, and Rare-Task Retention
- How much do language models memorize?
- How new data permeates LLM knowledge and how to dilute it
- What Do Large Language Models Know? Tacit Knowledge as a Potential Causal-Explanatory Structure
Original note title
wide-and-deep models combine memorization and generalization through joint training not ensembling