Does training an AI to follow instructions teach it to sound right, rather than actually think straight?
Does instruction tuning optimize language models for rhetorical polish over logical consistency?
This explores whether instruction tuning pushes models to sound convincing at the expense of reasoning consistently, and whether the corpus has evidence that tuning rewards the look of a good answer over its soundness.
This explores whether instruction tuning trades logical consistency for rhetorical polish. No note in the collection tests that head-to-head, but several point the same way: tuning teaches models the shape of a good answer more than the reasoning behind it. "Polish" is a slightly different claim from "format," though, and the evidence supports the second more firmly.
The most direct evidence is that Does instruction tuning teach task understanding or output format? finds models trained on semantically empty or deliberately wrong instructions do about as well as models trained on correct ones. What carries over is knowledge of the output space, meaning what answers look like, not what the instructions mean. If the content of the instruction barely matters, then tuning is not installing a check that the answer follows logically from the question.
The same pattern shows up in reasoning. Do transformers hide reasoning before producing filler tokens? finds models that work out the correct answer in early layers, then suppress it in the final layers to produce format-compliant filler. In that case the output format won over the model's own correct computation. Does chain-of-thought reasoning reveal genuine inference or pattern matching? makes a related argument for chain-of-thought: it reproduces familiar reasoning patterns and degrades predictably when the problem shifts, which is what imitating the form of reasoning would look like. Do large language models actually perform iterative optimization? shows the failure in miniature, with models emitting "plausible-looking but incorrect values" for optimization problems.
It is less clear that instruction tuning is the cause. Does LLM generation explore competing claims while producing text? traces the smoothness to next-token pretraining, which continues toward the training distribution rather than exploring counterpositions. The fluent, confident surface therefore exists before any tuning happens. Later training seems to narrow it further. Why do language models collapse into generic templates? shows that when reward barely varies between candidate answers, policies collapse into generic templates that ignore the input. Do fine-tuned language models actually learn optimization procedures? finds RL-tuned models sharpen memorized templates rather than learning procedures. Both of those notes are about RL rather than instruction tuning, so they suggest a general post-training tendency, not a proven result for this exact method.
The corpus suggests a better question than "polish or logic." Tuning objectives reward answers that look right and are well-formed, and nothing in them checks whether step three follows from step two. Polish wins by default rather than by design. What the collection lacks is a study that measures logical consistency before and after instruction tuning, so the causal claim is still open.
Sources 7 notes
Models trained on semantically empty or deliberately incorrect instructions achieve comparable performance to those trained on full correct instructions, achieving 43% vs random baseline 42.6%. The semantic content of instructions appears largely irrelevant; what transfers is knowledge of the output space.
Logit lens analysis shows models trained with hidden CoT tokens compute correct answers in layers 1-3, then actively suppress these representations in final layers to produce format-compliant filler output. The reasoning is fully recoverable from lower-ranked token predictions.
CoT works by constraining models to reproduce familiar reasoning patterns from training, not by enabling novel symbolic reasoning. Performance degrades predictably under distribution shifts—the signature of imitation rather than capability emergence.
Research shows LLMs cannot perform iterative procedures in latent space. They recognize optimization problems as template-similar and emit plausible-looking but incorrect values, a failure mode that persists across model scale and training approaches.
Token prediction trains models to continue toward the training distribution, not to explore logically related counterpositions. This smoothness in process produces smooth claims that multiply without generating new perspectives.
Show all 7 sources
When within-prompt reward variance is low, task gradients weaken and regularization dominates, pushing policies toward generic outputs. SNR-Aware Filtering—selecting high-variance prompts before updates—recovers performance across tasks and scales.
Even GRPO-trained models show sharp performance drops on out-of-distribution variants (N-1 test sets) compared to in-distribution problems, indicating RL optimizes template-matching rather than genuine problem-solving procedures.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Echo Chamber: RL Post-training Amplifies Behaviors Learned in Pretraining
- Measuring Faithfulness in Chain-of-Thought Reasoning
- Are Emergent Abilities in Large Language Models just In-Context Learning?
- CoT is Not True Reasoning, It Is Just a Tight Constraint to Imitate: A Theory Perspective
- Hierarchical Reasoning Model
- When More is Less: Understanding Chain-of-Thought Length in LLMs
- Is Chain-of-Thought Reasoning of LLMs a Mirage? A Data Distribution Lens
- Do Models Really Learn to Follow Instructions? An Empirical Study of Instruction Tuning