Why are rare tokens the hooks for verbatim model memorization?
This explores why low-frequency, surprising tokens — rather than common words — act as the anchors a model uses to recall a passage word-for-word, and what that reveals about how memorization is stored mechanically.
This explores why rare tokens become the hooks for verbatim recall — not because they're important to meaning, but because they're unpredictable, and unpredictability is exactly what a model can't reconstruct from its general grasp of language and so must store outright. The clearest mechanical evidence comes from work that traces memorized paragraphs to a specific fingerprint inside GPT-Neo: larger gradients in the lower layers and a single low-layer attention head that fixates on rare tokens, with recall hinging on just a few tokens near the start of the passage Where does a model store memorized paragraphs?. In other words, the rare token isn't incidental — it's the index. A common word like "the" appears everywhere and points nowhere in particular, but an unusual name, number, or string is nearly unique, so attending to it is a reliable way to locate one stored sequence and replay the rest.
Why rare tokens specifically carry this load becomes clearer when you look at how models treat frequency as a proxy for difficulty. Ordering training data from rare to common outperforms the usual easy-to-hard curriculum precisely because rarity signals a gap between the data and the pre-training distribution — the places where the model's smooth statistical generalization is weakest Does ordering training data by rarity actually improve language models?. Where generalization is weak, memorization fills in. The model can predict common continuations on the fly; rare ones it has to pin down, and pinning them down means storing a verbatim trace keyed on the surprising token.
The same theme shows up under a different name — entropy — in work that isn't about memorization at all. Byte-level models deliberately spend more compute on high-entropy (less predictable) regions of text and coast through predictable stretches Can byte-level models match tokenized performance with better efficiency?, and in reasoning models a small minority of high-entropy "forking" tokens turns out to carry most of the learning signal Do high-entropy tokens drive reasoning model improvements?. Rare tokens are simply the high-entropy points of ordinary text: the model's machinery naturally concentrates effort and storage exactly where the next token is hardest to guess. Verbatim memorization is what that concentration looks like when the "hard" content is a specific string rather than a reasoning decision.
Two more corners of the corpus sharpen the picture. Repetition turns this latent tendency into leakage — fine-tuning on repeated sensitive data drives memorization from near zero to 60–75%, and one of the defenses that works is entropy filtering, which targets the same surprising-token signature Does repeated sensitive data in fine-tuning cause memorization?. And memorization isn't confined to whole paragraphs: in chain-of-thought reasoning, local memorization keyed on immediately preceding tokens accounts for up to two-thirds of errors Where do memorization errors arise in chain-of-thought reasoning?, a reminder that token-anchored recall operates at every scale.
The thing you may not have expected to learn: rare tokens are hooks not despite being uninformative but because of it. Their statistical isolation — appearing in few contexts, hard to predict — is what makes them function as near-unique addresses. The model's drive to handle the unpredictable, visible as both a localized attention head and a general bias toward high-entropy regions, is the same mechanism that makes it leak training data verbatim. If you want to chase this further, the localization finding suggests memorization is targetable for surgical unlearning, while the entropy-filtering defense suggests it's detectable before it ever leaks.
Sources 6 notes
Memorized paragraphs leave a distinctive fingerprint in GPT-Neo: larger gradients in lower layers, concentration in a specific low-layer attention head attending to rare tokens, and dependence on a few early-prefix tokens. This localization makes memorization targetable for unlearning.
CTFT fine-tunes LLMs on rare data first because rarity signals distributional weakness, not conceptual difficulty. This reframes curriculum learning as managing distance from pre-training distribution rather than pedagogical scaffolding.
The Byte Latent Transformer (BLT) dynamically segments bytes into patches based on next-byte entropy, allocating more compute to high-entropy regions and less to predictable ones. At 8B parameters, BLT matches tokenized baselines while reducing inference cost and improving robustness to typos and cross-lingual transfer.
Only ~20% of tokens exhibit high entropy as pivotal reasoning decision points; RLVR primarily adjusts these forking tokens. Training exclusively on them matches or exceeds full-gradient performance, revealing that the minority carries the learning signal.
Controlled experiments on GPT-2, Phi-3, and Gemma-2 show fine-tuning with repeated sensitive data increases privacy leakage from baseline 0-5% to 60-75%. Four complementary defenses—semantic dedup, differential privacy, entropy filtering, and pattern filtering—eliminate leakage while preserving 94.7% utility.
STIM framework identifies local, mid-range, and long-range memorization sources in CoT reasoning. Local memorization—based on preceding tokens—accounts for up to 67% of reasoning errors, especially as complexity increases and distributional shift occurs.