SYNTHESIS NOTE
Can we predict keyword priming before learning happens?
Exploring whether the degree to which newly learned keywords contaminate unrelated contexts can be predicted from measurable properties before training begins, and what mechanisms enable this prediction.
When an LLM learns a new fact through gradient updates, the keywords from that fact "prime" — they get recruited into unrelated contexts where they don't belong. Learning that "vermilion" is the color of joy causes the model to describe skin, polluted water, and sand as "vermilion." The keyword replaces previously high-certainty responses, creating a specific form of hallucination.
The central finding: priming is predictable before learning. Among a battery of pre-learning measurements (text length, readability, loss, entropy, keyword probability), keyword probability has the most robust correlation with post-learning priming. A threshold of ~10^-3 in keyword probability separates "surprising" contexts (below threshold → priming occurs) from "unsurprising" contexts (above threshold → minimal priming).
This holds across:
- Different keyword sets
- Model sizes (PALM-2-XS, S)
- Architectures (PALM-2, Gemma, Llama) despite different backbones, training procedures, and data mixtures
- Training stages
The dynamics of contamination are concerning:
- Just 3 presentations of a single sample (even spaced every 20 minibatches) are sufficient to establish the priming relationship
- Two independent facts from different themes create independent priming effects without interference
- Priming is thematically bounded but not eliminated — cross-theme priming is attenuated but still present
Two mitigation techniques reduce priming 50-95% while preserving learning:
- Stepping-stone text augmentation — modifying the training text to reduce keyword surprise
- Ignore-k update pruning — pruning the most affected parameter updates
The practical implication: every gradient update is a potential contamination event. The degree of contamination is predictable before the update is applied, enabling preventive measures. This connects to How much poisoned training data survives safety alignment? — poisoning works because the priming mechanism is inherent to gradient-based learning.
Inquiring lines that read this note 54
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
How do training priors constrain what context information can override?- How do training-data priors influence model defaults when context is ambiguous?
- Why does training data saliency distort how models judge meaning?
- How does in-context learning trigger phase transitions in model behavior?
- Why does context information fail to override prior training associations?
- Why does keyword priming require only three training exposures to establish?
- Can priming from different facts interfere with each other in the same model?
- How much can mitigation techniques like augmentation reduce priming without harming learning?
- What mechanism makes keyword probability the strongest predictor of priming?
- How would you redesign context integration to prevent prior associations from dominating?
- How does keyword priming enable language models to spread poisoned information?
- Does foundational model training or user priors more strongly shape final outputs?
- How do model priors enable targeted context queries without full attention?
- Why do pretrained model priors reduce the usefulness of retrieved experience?
- Does attention bias explain grounding failure in language models?
- Can Q-priming further strengthen clarifying question behavior beyond social meta-learning alone?
- Can data filtering during pretraining prevent cognitive biases in language models?
- How do training associations override context information in language models?
- Can retrieval policies learn to use pretraining statistics as decision features?
- How does prompt iteration reinforce user bias without empirical anchoring?
- Can prompting unlock compositional skills that pretraining already learned?
- Do text-space skills transfer learning across different frontier models?
- Do few-shot examples improve in-context learning or add noise?
- Why does training data not function as a searchable corpus?
- Can document repetition accidentally memorize sensitive information instead of learning?
- Why does conceptual priming alone fail to produce consciousness claims?
- Does input surprise drive the implicit recognition of on-policy context?
- How does distributional shift toward rare inputs change memorization reliance?
- Does representational density emerge from training data exposure during pretraining?
- What distinguishes data that generalizes broadly from task-specific memorization?
- Why does semantic deduplication reduce memorization in fine-tuned models?
- How does representational density emerge from training data familiarity?
- Do sample-level similarities between pretraining and downstream tasks explain the frequency effect?
- Does latent density emerge during pretraining from training data familiarity?
Related concepts in this collection 5
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Concept map
Can we predict keyword priming before learning h…
How much poisoned training data survives safety al…
Why do language models ignore information in their…
Does training on AI-generated content permanently …
When do language models stop memorizing and start …
Can we prune training data without hurting model p…
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
your link
semantically near
linked from elsewhere
-
How much poisoned training data survives safety alignment?
Explores whether adversarial contamination at 0.1% of pretraining data can persist through post-training safety measures, and which attack types prove most resilient to alignment.
priming is the mechanism; poisoning exploits it; the 3-exposure finding explains why minimal poisoning data suffices
-
Why do language models ignore information in their context?
Explores why language models sometimes override contextual information with prior training associations, and whether providing more context can solve this problem.
priming creates new associations that can subsequently override context; the two mechanisms compound
-
Does training on AI-generated content permanently degrade model quality?
When generative models train on outputs from previous models, do the resulting models lose rare patterns permanently? The question matters because future training data will inevitably contain synthetic content.
priming and collapse are both consequences of how gradient updates reshape the model's internal distribution
-
When do language models stop memorizing and start generalizing?
Can we measure the exact capacity limit where models transition from memorizing training data to learning underlying patterns? Understanding this boundary could reshape how we think about model learning and privacy.
priming is a specific manifestation of how memorization consumes model capacity; the 3-exposure sufficiency finding maps to the low threshold at which capacity fills
-
Can we prune training data without hurting model performance?
This explores whether difficulty metrics can identify redundant training examples that can be safely removed. It matters because most datasets contain massive waste — if we can find which examples are truly necessary, we could train better models on far less data.
complementary perspectives on training data efficiency: pruning shows most data is redundant (easy examples removable), while priming shows even minimal data (3 exposures) can disproportionately affect generative behavior; the keyword probability threshold (~10^-3) functions as an implicit difficulty metric
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- How new data permeates LLM knowledge and how to dilute it
- Where to show Demos in Your Prompt: A Positional Bias of In-Context Learning
- Language models show human-like content effects on reasoning tasks
- Between Circuits and Chomsky: Pre-pretraining on Formal Languages Imparts Linguistic Biases
- Planted in Pretraining, Swayed by Finetuning: A Case Study on the Origins of Cognitive Biases in LLMs
- Echo Chamber: RL Post-training Amplifies Behaviors Learned in Pretraining
- Farther the Shift, Sparser the Representation: Analyzing OOD Mechanisms in LLMs
- Emergent Introspective Awareness in Large Language Models
Original note title
knowledge priming after gradient updates is predictable from keyword probability before learning — and just 3 exposures suffice