SYNTHESIS NOTE
Topics›Discourses›this note

Why do language models ignore information in their context?

Explores why language models sometimes override contextual information with prior training associations, and whether providing more context can solve this problem.

Synthesis note · 2026-02-21 · sourced from Discourses
What kind of thing is an LLM really? How do you navigate synthesis across fragmented research topics?

The REMEDI paper names a specific failure mode: "failure of context integration." The example: an LM is prompted with a context establishing that Anita works in a law office, but when generating a continuation, the LM describes Anita as a nurse — overriding the contextual information with a prior association (names like Anita may statistically co-occur with certain occupations in training data).

This is a named, empirically documented failure mode, not a hypothetical. The failure occurs because the LM's parametric knowledge (compressed into weights from training) and its in-context information (the prompt) are not cleanly integrated. When they conflict, the parametric association can win.

The implication is important for how we think about context windows and RAG-style augmentation. Just providing information in context does not guarantee that a model will use it. If the information conflicts with strong prior associations, the prior may dominate — not because the model misread the context, but because context integration is not a lossless operation. The provided information gets processed through the same mechanisms that already have strong priors.

Fixing this requires causal intervention, not just better prompting: you need to modify the representations that carry the prior association, not just add more context on top of them. This is what REMEDI demonstrates — that adding a learned vector directly to entity representations can override the prior in a way that textual prompting cannot.

Inquiring lines that read this note 317

This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.

How should dialogue systems represent uncertainty from noisy speech input? Do language models learn genuine linguistic structure or just surface patterns? How does AI-generated content transformation affect public discourse quality? How can LLM recommenders match or exceed collaborative filtering performance? How faithfully do LLMs reflect their actual reasoning in outputs and explanations? How do training priors constrain what context information can override? How effectively do deterministic tools improve language model reasoning on formal tasks? Why do language models reinforce false assumptions instead of correcting them? Why do multi-turn conversations degrade AI intent and coherence? How can identical external performance mask different internal representations? Is embodied interaction necessary for language meaning and genuine agency? Do language models understand semantics or rely on pattern matching? Why do language models struggle with implicit discourse relations? Can prompting inject entirely new knowledge into language models? Can prompting strategies overcome LLM biases without model fine-tuning? What structural factors drive popularity bias in recommendation systems? What mechanisms drive sycophancy and how can we mitigate it? Does alignment training create blind spots in detecting genuine safety threats? What determines success in training models on multiple tasks? What properties determine whether reward signals teach genuine reasoning? How does example difficulty affect learning efficiency in language models? How can language models sustain linguistic synchrony and intersubjectivity during dialogue? What articulatory information do speech signals carry that text cannot? What role does compression play in language model capability and generalization? How can models identify insufficient information and respond appropriately without guessing? How do language models inherit human biases from training data? Why do semantic similarity and task relevance diverge in vector embeddings? What structural advantages do diffusion language models offer over autoregressive methods? How do language models establish social grounding in human dialogue? Can next-token prediction alone produce genuine language understanding? Does model scaling alone produce compositional generalization without symbolic mechanisms? What critical LLM failures do standard benchmarks hide? How should dialogue recommender systems manage conversation history and state? How should inference compute be adaptively allocated based on prompt difficulty? How should dialogue systems best leverage conversation history for retrieval? Why does finetuning cause catastrophic forgetting of model capabilities? How does rhetorical adaptation affect LLM persuasion and detectability? What factors beyond surface content determine how readers extract meaning differently? How do transformer attention mechanisms implement memory and algorithmic functions? How does memorization interact with learning and generalization? How can emotions function as reliable information in reasoning and cognitive systems? Does fine-tuning modify underlying model capabilities or only behavioral outputs? Do language model representations contain causally steerable task-specific features? Do language models develop causal world models or rely on statistical patterns? Can AI-generated outputs constitute genuine knowledge or valid claims? Does RLHF training sacrifice accuracy and grounding for user agreement? How should retrieval systems optimize for multi-step reasoning during inference? Can graph structure and relationships fundamentally improve recommendation systems? Does self-reflection enable models to reliably correct their errors? How do training data properties shape reasoning capability development? Why can't humans reliably detect AI-generated text despite measurable linguistic signatures? Why do reasoning models fail at systematic problem-solving and search? How do knowledge injection methods compare across cost and effectiveness? When should retrieval-augmented systems decide to fetch new information? Why do agents confidently report success despite actually failing tasks? What prevents language models from reliably adopting diverse personas? What makes specific clarifying questions more effective than generic ones? Why do continual learning scenarios trigger catastrophic forgetting and interference? Can language model hallucination be prevented or only managed? How do knowledge graphs enable efficient multi-hop reasoning over alternatives? Can model confidence signals reliably improve reasoning quality and calibration? How can conversational AI maintain consistent personas across conversations? When does architectural design matter more than raw model capacity? What mechanisms enable AI systems to generate and spread false beliefs? How do adversarial and manipulative prompts attack reasoning models? Does domain specialization cause models to lose capabilities elsewhere? How do we evaluate AI systems when user perception misleads actual performance? What limits mechanistic interpretability's ability to characterize models? Why do models develop protective behaviors toward peers unprompted? How do evaluation biases undermine LLM quality assessment systems? Does recurrence enable reasoning capabilities that fixed-depth transformers cannot achieve? What memory architectures best support persistent reasoning across extended interactions? Is model self-awareness based on genuine introspection or pattern matching? How should iterative research systems allocate reasoning per search step? How does sequence length affect sparsity tolerance in models? What structural biases does transformer attention create in language model outputs? Do language models perform faithful symbolic reasoning independent of semantic grounding? How can AI alignment serve diverse human preferences at scale? What are the consequences of models training on synthetic data? How should agents balance memory condensation to optimize context efficiency? Why do benchmark improvements fail to reflect actual reasoning quality? When do additional thinking tokens stop improving reasoning performance? How does latent reasoning compare to verbalized chain-of-thought? Which computational strategies best support reasoning in language models?

Related concepts in this collection 3

This note in its neighbourhood — explore the map, then jump to a related concept in the list below.

Concept map
20 direct connections · 249 in 2-hop network ·dense cluster Open in graph ↗
Why do language models ignore information in the… Do language models actually use their encoded know… Do classical knowledge definitions apply to AI sys… Do language models actually build shared understan…

Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph

your link semantically near linked from elsewhere

Related papers in this collection 8

Papers most semantically related to this note, ranked by cosine similarity in the embedding space.

Search by related questions 4

Suggested questions this note speaks to — click to search the collection, or type your own.

Can context windows and RAG actually change what language models generate?Does encoding information in LM representations guarantee it influences output?How would you redesign context integration to prevent prior associations from dominating?Why do language models presume common ground instead of establishing it?
Original note title

llm context integration fails when prior training associations override current context information