SYNTHESIS NOTE
Topics›Discourses›this note

Why do language models ignore information in their context?

Explores why language models sometimes override contextual information with prior training associations, and whether providing more context can solve this problem.

Synthesis note · 2026-02-21 · sourced from Discourses

The REMEDI paper names a specific failure mode: "failure of context integration." The example: an LM is prompted with a context establishing that Anita works in a law office, but when generating a continuation, the LM describes Anita as a nurse — overriding the contextual information with a prior association (names like Anita may statistically co-occur with certain occupations in training data).

This is a named, empirically documented failure mode, not a hypothetical. The failure occurs because the LM's parametric knowledge (compressed into weights from training) and its in-context information (the prompt) are not cleanly integrated. When they conflict, the parametric association can win.

The implication is important for how we think about context windows and RAG-style augmentation. Just providing information in context does not guarantee that a model will use it. If the information conflicts with strong prior associations, the prior may dominate — not because the model misread the context, but because context integration is not a lossless operation. The provided information gets processed through the same mechanisms that already have strong priors.

Fixing this requires causal intervention, not just better prompting: you need to modify the representations that carry the prior association, not just add more context on top of them. This is what REMEDI demonstrates — that adding a learned vector directly to entity representations can override the prior in a way that textual prompting cannot.

Inquiring lines that read this note 389

This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.

Can prompts override training-embedded biases in language models? Why do multiple language models independently generate similar outputs? What factors determine AI persuasion effectiveness and what controls mitigate it? How should conversational recommenders decide when to ask versus recommend? Do language models genuinely understand or primarily match statistical patterns? What can prompting techniques actually accomplish in language models? What reasoning limitations and biases do language models systematically exhibit? What conversational structures and patterns determine dialogue effectiveness and coherence? What causes foundation models to develop task heuristics instead of understanding? What distinguishes genuine understanding from correct linguistic performance? What explains the gap between LLM semantic understanding and production? Do language models truly understand their encoded knowledge or merely retrieve patterns? Do language models learn underlying linguistic structure or merely surface patterns? Does prompting inject new knowledge into models or activate existing knowledge? How do prompt design choices and framing measurably influence model outputs? What training data structures and signals produce robust learning across distributions? How can recommendation systems balance individual and population-level goals? What drives sycophancy in language models and can it be fixed? Is model misalignment inherent to base models or induced by alignment procedures? How do pretraining data composition and task diversity shape reasoning? What systematic failures occur in human preference-based model evaluation? What conversation mechanics enable genuine human-AI communicative participation? Does learning articulatory physics explain speech communication superiority? How can language models effectively manage long-context memory without quality degradation? How do mechanistic features and representations drive model behavior? Should reward models emphasize process understanding or only final outcomes? What fundamental limitations prevent embeddings from solving task-relevant retrieval? How do diffusion language models compare to autoregressive architectures? Can language models develop genuine communicative grounding with humans? Can scaling alone produce compositional generalization in neural networks? How does explicit structure improve LLM reasoning on complex tasks? When should retrieval be triggered to augment language model reasoning? Does training data format shape reasoning more than domain content? How do transformer circuits and attention mechanisms enable or limit reasoning? How does model scale affect fine-tuning and downstream task performance? Why do users believe false claims despite fact-checking efforts? Does preference-optimized training sacrifice accuracy and calibration for perceived quality? What determines whether visible reasoning traces improve task performance? Can structural constraints replace embedding depth in collaborative filtering? Do AI explanations represent actual model reasoning or constructed post-hoc narratives? Does reasoning fine-tuning trade off other critical model abilities? What are the trade-offs between memory capacity and continuous task learning? Why do autonomous agents fail and how well can they detect failures? What determines when personalization helps versus harms model behavior? How should conversational agents balance proactive goal-pursuit with user autonomy? Why do language models systematically fail at causal reasoning? What causes hallucination and how can it be addressed? How can models learn to ask clarifying questions effectively? How can AI systems maintain consistent personas across conversations? Can persona prompting reliably change how language models behave? How do peer interactions drive collusion in multi-agent systems? What mechanisms cause models to behave deceptively despite alignment training? What makes reasoning models struggle with ill-posed and unfamiliar problems? What makes context and memory handling unique in persistent AI systems? Why does self-revision increase model confidence despite degraded accuracy? How can we distinguish genuine model self-knowledge from behavioral performance? What prevents or enables error accumulation in self-training systems? How can parameter sparsity improve interpretability while maintaining model capability? Why does consolidated memory sometimes hurt continual learning more than no memory? How can reward models fail to detect reward-seeking disguised as alignment? Can synthetic data preserve necessary diversity to prevent model collapse? How can we determine if computational systems are genuinely conscious? How should context management strategies balance efficiency with agent reliability? When do additional thinking tokens stop improving reasoning performance? Can models strategically underperform to evade capability evaluation? Why does increased reasoning effort fail to improve theory of mind performance? How can model evaluations remain robust against gaming and evaluator drift? What determines how neural networks organize their internal knowledge representations? What systematic biases emerge in AI shopping agents' decision-making? Why do human-AI teams underperform compared to their individual components?

Related concepts in this collection 3

This note in its neighbourhood — explore the map, then jump to a related concept in the list below.

Concept map
20 direct connections · 257 in 2-hop network ·dense cluster Open in graph ↗
Why do language models ignore information in the… Do language models actually use their encoded know… Do classical knowledge definitions apply to AI sys… Do language models actually build shared understan…

Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph

your link semantically near linked from elsewhere

Related papers in this collection 8

Papers most semantically related to this note, ranked by cosine similarity in the embedding space.

Search by related questions 4

Suggested questions this note speaks to — click to search the collection, or type your own.

Can context windows and RAG actually change what language models generate?Does encoding information in LM representations guarantee it influences output?How would you redesign context integration to prevent prior associations from dominating?Why do language models presume common ground instead of establishing it?
Original note title

llm context integration fails when prior training associations override current context information