When do language models stop memorizing and start generalizing?
Can we measure the exact capacity limit where models transition from memorizing training data to learning underlying patterns? Understanding this boundary could reshape how we think about model learning and privacy.
The standard approach to measuring memorization — attempting to extract training data from the model — is fundamentally flawed. Language models can be coerced to output almost any string, so generation is not proof of memorization. Conversely, a model may memorize patterns (every other token, structural regularities) without reproducing text verbatim. Extraction is neither necessary nor sufficient.
The formal separation: unintended memorization is the information a model contains about a specific dataset (the bits that would change if a particular example were removed from training). Generalization is the information the model contains about the true data-generation process. By isolating and eliminating the generalization component, total memorization becomes measurable.
The key empirical finding: GPT-family models have an approximate capacity of 3.6 bits-per-parameter for unintended memorization. Models memorize training data until this capacity fills. At that point, a phase transition occurs — grokking begins, and unintended memorization decreases as models begin to generalize.
This reframes the grokking phenomenon mechanistically. Since What happens inside models when they suddenly generalize?, the capacity-filling measurement adds the trigger condition: grokking doesn't begin at an arbitrary training step — it begins when memorization saturates. The three phases are downstream of a capacity constraint, not of training duration per se.
The practical implication: memorization capacity is a measurable property of a specific model, not a property of the training algorithm. Two models trained by the same algorithm on the same data can have different memorization properties. This matters for privacy (which models leak more), for understanding generalization (capacity constrains when it begins), and for the Can AI pass every test while understanding nothing? question — a model that appears to generalize may simply have unfilled memorization capacity.
Inquiring lines that read this note 31
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
How do training priors constrain what context information can override? How does reasoning graph topology affect breakthrough insights and generalization? How does memorization interact with learning and generalization?- How much does memorization capacity limit a model's ability to learn new information?
- Why is extracting training data insufficient proof that models memorize?
- What is the theoretical capacity limit before memorization saturates?
- How does memorization capacity saturation trigger the grokking transition?
- How do the three grokking phases connect to memorization capacity limits?
- Can data pruning strategies exploit the finite nature of memorization capacity?
- Do models with unfilled memorization capacity appear to generalize falsely?
- How do overparameterization and data size shift what attractors represent?
- What makes data augmentation an implicit form of contraction learning?
- What distinguishes data that generalizes broadly from task-specific memorization?
- Does grokking in modular arithmetic follow the same three-phase learning trajectory?
- How do models develop dense representations for familiar training data?
- What determines whether accumulated state generalizes spuriously across continual learning domains?
- Why does grokking reveal the shift from memorization to genuine understanding?
- How do retention gates regularize forgetting across different sequence model architectures?
- What capacity limits does the memory model face as corpus grows?
- How does modeling capability relate to lossless compression in language models?
- Can model compression size predict generalization better than parameter count?
- Can compression length really indicate how well a model generalizes?
Related concepts in this collection 3
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
-
What happens inside models when they suddenly generalize?
Grokking appears as an abrupt shift from memorization to generalization. But is the underlying process truly discontinuous, or does mechanistic analysis reveal continuous phases we can measure and predict?
capacity-filling provides the trigger mechanism for when grokking begins
-
Can we predict keyword priming before learning happens?
Exploring whether the degree to which newly learned keywords contaminate unrelated contexts can be predicted from measurable properties before training begins, and what mechanisms enable this prediction.
a complementary view of how memorization interacts with learning
-
Can we prune training data without hurting model performance?
This explores whether difficulty metrics can identify redundant training examples that can be safely removed. It matters because most datasets contain massive waste — if we can find which examples are truly necessary, we could train better models on far less data.
if memorization has finite capacity, pruning removes low-value items that consume capacity
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- How much do language models memorize?
- Localizing Paragraph Memorization in Language Models
- Provable Benefits of In-Tool Learning for Large Language Models
- Diagnosing Memorization in Chain-of-Thought Reasoning, One Token at a Time
- Repeat After Me: Transformers are Better than State Space Models at Copying
- SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-training
- Towards Mechanistically Understanding Why Memorized Knowledge Fails to Generalize in Large Language Model Finetuning
- How new data permeates LLM knowledge and how to dilute it
Original note title
llm memorization formally separates into unintended memorization and generalization — 3.6 bits-per-parameter capacity fills before grokking begins