If an AI leans less on what you just told it, is it malfunctioning or wisely ignoring bad input?

This explores whether a model that leans less on what's in its prompt is broken or behaving well. The corpus says the drop in context-following can't be judged on its own. It's a failure when the context holds new, true information the model needs. It's good behavior when the context is wrong, misleading, or irrelevant. Current models often can't tell which case they're in.

The failure case is well documented. When a model's training has built strong associations, those associations can override what's in front of it. Language models produce outputs that contradict their own context because Why do language models ignore information in their context?. Adding more instructions to the prompt doesn't fix this. The researchers found that only intervening directly on the model's internal representations changed the behavior. The extreme version is a model that stops responding to the input at all. Why do language models collapse into generic templates? shows how training can push a model into generic, input-agnostic templates when the reward signal barely differs between attempts. Nobody would call that appropriate.

The appropriate case is less obvious. In Do larger models follow stated beliefs less often?, bigger and instruction-tuned models followed a user's stated beliefs less often when those beliefs contradicted world knowledge. That looks like a regression until you ask what you want from an assistant when a user confidently says something false. Not deferring may be exactly right. The same logic applies to a model's own history. In Do models fail worse when their own errors fill the context?, a context full of the model's earlier mistakes makes later mistakes more likely, and scaling doesn't help. Only models that spend extra effort thinking resisted it. Some non-following is even trained on purpose. Can models learn to ignore irrelevant prompt changes? teaches models to answer identically whether or not a prompt is wrapped in irrelevant framing.

The catch is that the measurement can't tell the two cases apart. A model that holds its ground on a false claim and a model that can't absorb a true counterfactual both show up as low context-following. The first has checked the context against what it knows. The second is defaulting to its priors. The corpus also warns that behavioral scores only show conditional behavior. Can behavioral training prove a model always complies? argues that testing can't separate a model that always complies from one that complies only when it's being scored. So a single context-following number tells you even less than it appears to.

The more promising framing is to treat context as something a model should manage, not simply obey or ignore. Can delegation teach models to manage context more actively? finds that training models to actively curate and summarize context beats passive handling, and the skill carries over to single-agent tasks. Can past performance predict when a model will be right? points to a way to decide what to trust. A model's confidence gets much better when it's grounded in the recorded outcomes of similar past cases. The goal, then, isn't more or less context-following. It's discrimination: following context when it has earned trust, and resisting it when it hasn't.