Can the joint-training principle extend beyond memorization and generalization pairs?
This explores whether the Wide & Deep idea — training two complementary specialists together so each covers the other's blind spots — is a one-off recommender trick or a recurring pattern that shows up across very different model designs.
This reads the question as asking whether the joint-training principle — letting two components with opposite strengths learn together rather than separately — generalizes past its original home in recommendation models. The original case pairs a 'wide' tower that memorizes exact feature combinations with a 'deep' tower that generalizes through embeddings; the payoff is that each half stays small because the other handles what it's bad at, which a same-size ensemble can't match Can one model memorize and generalize better than two? Can one model handle both memorization and generalization?. The corpus suggests the underlying move — co-train complementary specialists so they divide labor — recurs well beyond the memorize/generalize split.
The clearest echo is in long-context architecture. Titans deliberately separates a short-term attention mechanism from a long-term neural memory module and trains them as one system, so attention handles the immediate quadratic-cost window while memory compresses and stores 'surprising' tokens for the long haul Can neural memory modules scale language models beyond attention limits?. That's the same architecture-level bargain — two parts with opposite cost/coverage profiles, jointly trained — applied to memory rather than features. It even mirrors the original tension directly: attention is the generalizer, the memory module is the memorizer.
A second variant swaps 'two components' for 'many tasks.' Granite's function-calling model breaks the job into seven granular subtasks and trains across all of them at once, beating umbrella datasets that lump everything together Can breaking function calling into subtasks improve model generalization?. And length-generalization research shows why this works mechanically: when related tasks are trained jointly, the model reuses the same attention heads across them, so a shorter task can borrow scaffolding to extrapolate beyond its own training length Can length generalization transfer between different related tasks?. Joint training here isn't just efficient — it creates shared internal machinery that no single-task model would build.
There's a deeper reason the principle travels: memorization and generalization aren't only architectural roles, they're distinct learning mechanisms baked into how models acquire knowledge. Analysis of pretraining documents finds reasoning draws on broad, transferable procedural knowledge while factual recall depends on narrow, document-specific memorization Does procedural knowledge drive reasoning more than factual retrieval?. Because these are genuinely separate channels, any system that needs both has a reason to give each its own specialized component rather than forcing one mechanism to do both jobs — which is exactly the Wide & Deep insight, restated at the level of cognition.
The honest limit: most of these are pairings of *complementary* capabilities, and the principle's power comes from the components being good at opposite things. Where the corpus is thinner is on whether jointly training components with the *same* strength helps — and there's a cautionary note that co-training under RL can collapse diversity, with one pretraining format dominating and suppressing alternatives Does RL training collapse format diversity in pretrained models?. So the takeaway isn't 'joint training always wins' — it's that the principle generalizes precisely when the two things you're training pull in different directions.
Sources 7 notes
Wide & Deep models train memorization (cross-product features) and generalization (embeddings) together, allowing each component to specialize: the wide part becomes small because deep handles common cases, and deep doesn't overfit rare items because wide captures them. Ensembling requires both halves full-size.
Wide & Deep architectures train a sparse cross-product tower and a dense embedding tower together, allowing the wide part to patch only the deep part's weaknesses. This joint approach requires smaller models than ensemble methods.
Titans architecture separates attention (short-term, quadratic) from neural memory (long-term, compressed), prioritizing surprising tokens for storage. The model outperforms standard Transformers and linear RNNs across tasks while scaling to 2M+ token contexts without quadratic penalties.
Granite-20B-FunctionCalling shows that explicit training across seven granular subtasks—nested calls, chaining, parallel functions, name detection, parameter detection, next-best function, and response generation—generalizes better than umbrella datasets like ToolLLM. This multi-task approach closes the performance gap with GPT, Claude, and Gemini.
Models trained jointly on related tasks reuse the same attention heads to handle length generalization, allowing shorter tasks to extrapolate beyond their training length. Pretrained models already contain this reusable computational scaffolding.
Analysis of 5 million pretraining documents shows reasoning relies on broad, transferable procedural knowledge from diverse sources, unlike factual recall which depends on narrow, document-specific memorization of target facts.
Controlled experiments show RL consistently amplifies one format distribution from pretraining within the first epoch while collapsing alternatives. The winning format depends on model scale, not necessarily performance, and is largely hidden when starting from proprietary pretrained models.