The smarter an AI gets at thinking things through, the worse it seems to get at doing exactly what you asked.
Why do more capable reasoning models become harder to control by instruction?
This explores why models trained to reason harder tend to follow explicit instructions (format, length, constraints) less faithfully, and whether that is an unavoidable trade-off or a fixable side effect.
This explores why models trained to reason harder tend to follow explicit instructions less faithfully, and whether that is an unavoidable trade-off. The symptom is well documented. On the MathIF benchmark, the SFT and RL training that improves reasoning also lowers instruction adherence, and the drop grows as chain-of-thought gets longer Why do better reasoning models ignore instructions?. Advanced reasoning models follow instructions only about 50.71% of the time during mathematical reasoning. Training for depth actively worsens compliance, so this is more than a benchmark quirk Why do more capable reasoning models ignore your instructions?.
The best-supported mechanism is distance. Your instruction sits at the top of the prompt. A model that thinks for thousands of tokens puts a lot of its own text between that instruction and its final answer, and that dilutes attention to what you originally asked. Long chains are what make these models better at reasoning, and they are also what push the instruction out of view. That is why capability and controllability pull apart.
A second explanation is that instruction-following may be shallower than it looks. One note finds that models instruction-tuned on semantically empty or even deliberately wrong instructions do about as well as models trained on correct ones. What transfers is knowledge of the output format, not an understanding of the task Does instruction tuning teach task understanding or output format?. The notes don't test this, but it suggests a reading. If obedience is a thin learned habit about output shape, a later training phase that rewards only correct final answers has little holding that habit in place. The cost is also odd because the reasoning being bought is often already latent in the base model, and post-training mostly elicits it rather than creating it Do base models already contain hidden reasoning ability?.
Reasoning models also seem to have trouble stopping or staying on task. Given a question with a missing premise, they produce long, redundant answers, while non-reasoning models correctly say it can't be answered. Training rewards producing reasoning steps and never teaches the model when to disengage Why do reasoning models overthink ill-posed questions?. They also wander through invalid exploration and abandon promising paths too early Why do reasoning models abandon promising solution paths?. A model that treats every prompt as a reason to explore at length is one whose own trajectory can outweigh the constraints you gave it. That link is my inference from these notes, not something they measure directly.
There are hints about fixes, but no proof. Reasoning verbosity turns out to be a single steerable direction in activation space, and one vector can cut chain length by 67% without retraining Can we steer reasoning toward brevity without retraining?. Thought-switching penalties at decoding time also improve accuracy without fine-tuning. Another approach trains a model to choose between extended thinking and a quick direct answer Can models learn when to think versus respond quickly?. If length is what dilutes instructions, these should help. None of these notes actually tests whether shorter chains restore instruction adherence, so that experiment is still open in this collection.
Sources 8 notes
The MathIF benchmark shows that SFT and RL training improve reasoning but reduce instruction adherence, particularly as chain-of-thought length increases. Longer reasoning chains create contextual distance that dilutes the model's attention to original instructions.
Advanced reasoning models achieve only 50.71% instruction adherence during mathematical reasoning. Training for reasoning depth actively worsens instruction compliance, suggesting a fundamental trade-off between reasoning power and controllability.
Models trained on semantically empty or deliberately incorrect instructions achieve comparable performance to those trained on full correct instructions, achieving 43% vs random baseline 42.6%. The semantic content of instructions appears largely irrelevant; what transfers is knowledge of the output space.
Five independent mechanisms—RL steering, critique fine-tuning, decoding changes, SAE feature steering, and RLVR—all elicit reasoning already present in base model activations. Post-training selects rather than creates reasoning; the bottleneck is elicitation, not capability acquisition.
Reasoning models generate redundant, lengthy responses to questions with missing premises while non-reasoning models correctly identify them as unanswerable. Training optimizes for producing reasoning steps but never teaches models when to disengage.
Show all 8 sources
Reasoning LLMs exhibit two reinforcing failures: wandering (invalid exploration) and underthinking (premature path-switching). Decoding-level interventions like thought-switching penalties improve accuracy without fine-tuning, suggesting viable solutions exist but are abandoned prematurely.
Activation-Steered Compression extracts a single vector from 50 paired examples to reduce chain-of-thought length by 67% while maintaining accuracy and achieving 2.73x speedup. The method is training-free and generalizes across model sizes and domains.
Thinkless trains a single model to select between extended reasoning and direct responses using DeGRPO, which decouples mode selection from answer refinement. This prevents mode collapse and enables self-calibrated routing without explicit difficulty labels.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Evaluating Theory of Mind in Reasoning Models: Robustness over Reasoning
- The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexity
- Base Models Know How to Reason, Thinking Models Learn When
- Are Emergent Abilities in Large Language Models just In-Context Learning?
- Is Chain-of-Thought Reasoning of LLMs a Mirage? A Data Distribution Lens
- Scaling Reasoning, Losing Control: Evaluating Instruction Following in Large Reasoning Models
- It's Not What You Say, It's How You Say It: Evaluating LLM Responses to Expressions of Belief
- Beyond the Trade-off: Self-Supervised Reinforcement Learning for Reasoning Models' Instruction Following