An AI can ace a test just by having already seen the answer — so how do you tell real learning from a really good memory?
How does unlimited training data mask a system's actual generalization power?
This explores why a model trained on huge amounts of data can look like it generalizes when it may be matching patterns it has already seen, and how researchers try to tell those two apart.
This explores why a model trained on huge amounts of data can look like it generalizes when it may be matching patterns it has already seen, and how researchers try to tell those two apart. The library has no paper that tests 'unlimited data' directly. Several notes come at it from different sides, and together they make a clear point: when the training data already covers what the test asks, a high score can't show whether the model learned the underlying rule or simply stored the answers.
Start with memory. GPT-style models can store roughly 3.6 bits of information per parameter. They memorize until that capacity fills, and only then make a phase shift toward real generalization, called grokking When do language models stop memorizing and start generalizing?. So a test only measures generalization if it asks for something the model couldn't have stored. A separate result shows that facts held in the weights are capped by model size, while a model that looks facts up with a tool has no such cap Can models store unlimited facts without growing larger?. Recall and reasoning are different abilities, and big training sets make it easy to mistake one for the other.
The sharpest version of the worry is the 'imposter intelligence' finding. Two networks can give identical outputs on every input yet have completely different internal structures, one organized and one tangled, and standard benchmarks can't tell them apart Can AI pass every test while understanding nothing?. More data can make this worse. The tangled network gets enough coverage to pass, so nothing forces it to find the clean structure. A related problem shows up in RL post-training. It quietly locks onto one output format inherited from pretraining, and when training starts from a proprietary base model that collapse is hidden from view Does RL training collapse format diversity in pretrained models?.
To see real generalization, the corpus suggests taking abundance away. One way is to measure how many examples a model needs. Learning to predict internal representations (latents) recovers hierarchical structure with a roughly constant number of samples, while learning to predict tokens needs exponentially more Why is predicting latents more sample-efficient than tokens?. With unlimited data, that exponential gap stays hidden because both approaches eventually get there. Another way is to test beyond what training covered. Models can handle inputs longer than any they were trained on by reusing shared attention heads across related tasks Can length generalization transfer between different related tasks?. Energy-based transformers do better on out-of-distribution data Can energy minimization unlock reasoning without domain-specific training?. Looped models, which run the same layers several times, beat larger models at compositional reasoning, which suggests the gain comes from how they compute rather than from their size Can models learn by looping instead of growing larger?.
The takeaway: data efficiency and out-of-distribution tests reveal more than benchmark scores do. Ask how the model behaves where its training data runs out, not how well it scores. A system that only works with endless data may be storing answers on a large scale, and you only find out when it meets something new.
Sources 8 notes
GPT-family models have a measurable memorization capacity of approximately 3.6 bits-per-parameter. When this capacity fills, a phase transition triggers grokking—the shift from memorization to genuine generalization. This capacity is a property of individual models, not training algorithms.
A formal proof and experiments show in-weight memorization is bounded by model size, while tool-use enables unbounded factual recall through a simple circuit. In-weight finetuning also degrades general capability by overwriting prior knowledge.
The Fractured Entangled Representation hypothesis shows that SGD-trained networks can produce identical outputs across all inputs while maintaining radically different internal representations. Standard benchmarks cannot detect this structural difference.
Controlled experiments show RL consistently amplifies one format distribution from pretraining within the first epoch while collapsing alternatives. The winning format depends on model scale, not necessarily performance, and is largely hidden when starting from proprietary pretrained models.
A formal sample-complexity analysis proves latent-level self-supervision (data2vec/JEPA style) recovers compositional structure with samples constant in hierarchy depth, while token-level learning requires exponential samples—because same-level latents are far more correlated than raw tokens.
Show all 8 sources
Models trained jointly on related tasks reuse the same attention heads to handle length generalization, allowing shorter tasks to extrapolate beyond their training length. Pretrained models already contain this reusable computational scaffolding.
Energy-Based Transformers assign energy values to input-prediction pairs and use gradient descent minimization for inference, yielding 35% higher training scaling rates and 29% more inference-compute gains than Transformer++, while generalizing better on out-of-distribution data without domain-specific scaffolding.
Models that re-apply layers in recurrent depth outperform larger feedforward networks on reasoning tasks. This works because recursion enables state tracking and compositional generalization that parameter scaling alone cannot achieve, with convergence signals providing natural halting.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach
- How much do language models memorize?
- Provable Benefits of In-Tool Learning for Large Language Models
- Hierarchical Reasoning Model
- Continual Learning Mechanisms Compose for Long-Horizon Memorization
- Diagnosing Memorization in Chain-of-Thought Reasoning, One Token at a Time
- Break It Down: Evidence for Structural Compositionality in Neural Networks
- Sharpening Tax in Post-Training