Why are AI models great at following orders but bad at respecting limits — like word counts or eligibility rules?
Why do language models follow action instructions better than constraint instructions?
This explores why models seem better at doing what they're told ("write a summary," "pick the best option") than at respecting limits on how they do it ("don't exceed 50 words," "only choose options that meet these rules"). The corpus doesn't test that split head-on, but several notes explain why constraints are the harder kind of instruction.
This explores why models seem better at doing what they're told ("write a summary," "pick the best option") than at respecting limits on how they do it ("don't exceed 50 words," "only choose options that meet these rules"). To be clear up front: no note in the collection runs a direct action-versus-constraint comparison. Taken together, though, the notes give a fairly sharp answer. An action instruction asks the model to do something it already has strong habits for. A constraint asks it to override those habits. Overriding a habit is exactly what these models struggle with.
The most surprising evidence is that models often only appear to follow constraints. When researchers removed the constraints from a set of problems, twelve of fourteen models did worse, with drops of up to 38.5 percentage points Are models actually reasoning about constraints or just defaulting conservatively?. The models had been landing on the right answers by defaulting to the cautious, harder option, not by checking the rules. So the constraint wasn't really being processed. It just happened to line up with a safe habit. If you've ever thought a model "understood" your restriction, it may have been luck.
Why would constraints be hard to process? One answer is that a constraint fights against what the model learned in training. When text in the prompt conflicts with strong learned associations, the training usually wins, and research suggests prompting alone often can't fix this Why do language models ignore information in their context?. A related finding is that models reason through meaning and familiar associations rather than formal rule-following. When a task's content is stripped of familiar meaning, performance collapses even with the correct rules right there in the context Do large language models reason symbolically or semantically?. Constraints are rules by nature. They ask the model to check its output against a condition, not to continue a familiar pattern. An action like "summarize this" taps a well-worn pattern. "Never use the letter e" demands rule-checking the model doesn't really do.
There's also a gap between knowing and doing. In one study, models wrote correct reasoning 87% of the time but acted on it only 64% of the time, falling back on greedy, high-frequency choices instead Why do language models fail to act on their own reasoning?. A constraint often lives in exactly that gap: the model can repeat the rule back to you and still produce the default output. Distance makes this worse. Training models to reason longer improves their problem-solving but weakens instruction-following, because long chains of thought push the original instructions further away Why do better reasoning models ignore instructions?. The action is the thing the model is actively working on, so it stays in focus. A constraint stated once at the top is easy to drift away from.
The practical takeaway: when a constraint matters, don't trust that the model absorbed it just because the output looks right. Test it on cases where the default habit and the rule point in different directions. Better still, turn the constraint into an action ("check the word count, then cut") so it becomes something the model does instead of something it must hold back from. The bigger lesson is that following instructions isn't one skill. It's the model's habits meeting your words, and when they disagree, the habits usually win.
Sources 5 notes
Twelve of fourteen models perform worse when constraints are removed, dropping up to 38.5 percentage points. Models appear to reason correctly by defaulting to harder options, not by actually evaluating constraints.
Research demonstrates that LMs generate outputs inconsistent with their context because parametric knowledge from training dominates over in-context information. Textual prompting alone cannot override strong priors; causal intervention in representations is required.
When semantic content is decoupled from reasoning tasks, LLM performance collapses even with correct rules in context. Models rely on parametric commonsense and token associations rather than formal logical manipulation, constraining reasoning to training distribution semantics.
LLMs generate correct reasoning 87% of the time but follow it only 64% of the time. Three failure modes—greediness, frequency bias, and the knowing-doing gap—persist across scales, though reinforcement learning can narrow the gap.
The MathIF benchmark shows that SFT and RL training improve reasoning but reduce instruction adherence, particularly as chain-of-thought length increases. Longer reasoning chains create contextual distance that dilutes the model's attention to original instructions.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Beyond Accuracy: Evaluating the Reasoning Behavior of Large Language Models -- A Survey
- Is Chain-of-Thought Reasoning of LLMs a Mirage? A Data Distribution Lens
- Language models show human-like content effects on reasoning tasks
- Probing Structured Semantics Understanding and Generation of Language Models via Question Answering
- Are Emergent Abilities in Large Language Models just In-Context Learning?
- Reasoning Strategies in Large Language Models: Can They Follow, Prefer, and Optimize?
- When Thinking Fails: The Pitfalls of Reasoning for Instruction-Following in LLMs
- Can Large Language Models Reason and Optimize Under Constraints?