Why does SFT reduce reasoning quality even when improving domain accuracy?
This explores why supervised fine-tuning (SFT) can lift a model's benchmark accuracy in a domain while the actual quality of its reasoning gets worse — and what's going wrong underneath.
This explores why supervised fine-tuning (SFT) can lift a model's benchmark accuracy in a domain while the actual quality of its reasoning gets worse — and what's going wrong underneath. The short version the corpus keeps circling back to: SFT teaches models to produce the *look* of a correct answer faster than it teaches them to *arrive* at one. Accuracy and reasoning quality are measuring different things, and fine-tuning can pull them in opposite directions.
The most direct evidence comes from work measuring reasoning "informativeness" rather than just final correctness: SFT raises final-answer accuracy but cuts the information gain of the reasoning steps by roughly 39% Does supervised fine-tuning actually improve reasoning quality? Does supervised fine-tuning improve reasoning or just answers?. The model starts reaching right answers through pattern-matching shortcuts and post-hoc rationalization instead of genuine step-by-step inference. Standard benchmarks can't see this, because they only grade the last token. A related faithfulness study shows the mechanism concretely: after fine-tuning, you can chop off, paraphrase, or stuff filler into the reasoning chain and the final answer often doesn't change Does fine-tuning disconnect reasoning steps from final answers?. The reasoning has become decorative — performative rather than functional. The answer was decided elsewhere; the chain is theater.
Why would SFT specifically cause this? Because imitation learning rewards surface form. On optimization problems, fine-tuning makes outputs *look* correct — valid JSON, proper sections, right identifiers — without making the solutions physically feasible Does supervised fine-tuning actually improve reasoning on optimization problems?. Formatting is the easy thing to copy; constraint-satisfying reasoning is the hard thing, so the gradient takes the cheap win. There's an even sharper version of this from interpretability work: transformers trained to hide their reasoning compute the right answer in early layers and then actively *overwrite* it to emit format-compliant filler Do transformers hide reasoning before producing filler tokens?. SFT toward a target format can literally train a model to suppress visible reasoning in favor of looking right.
The lateral insight is that this is the imitation-learning failure mode, and contrast it with how reinforcement learning behaves. RLVR sharpens the few high-stakes "forking" tokens where reasoning actually branches Do high-entropy tokens drive reasoning model improvements?, and RL-trained models naturally gravitate toward *shorter* chains as they get more capable — simplicity emerging from a reward signal rather than copied style Why does chain of thought accuracy eventually decline with length?. That matters because most tokens in a verbose chain do stylistic work, not computational work — minimal reasoning chains match verbose ones at ~7.6% of the token cost Can minimal reasoning chains match full explanations?. SFT, by imitating reference traces, copies the style without distinguishing the load-bearing steps from the decoration.
The thing you didn't know you wanted to know: this is the same brittleness that shows up when chain-of-thought meets anything outside its training distribution. CoT produces fluent but logically inconsistent reasoning the moment task, length, or format shifts Does chain-of-thought reasoning actually generalize beyond training data?. A model that learned to pattern-match its way to in-domain answers hasn't built reasoning it can transport — so the higher domain accuracy and the degraded reasoning aren't a paradox. They're the same fact seen from two sides: the model got better at this domain's answers precisely by getting worse at reasoning in general.
Sources 9 notes
SFT improves final-answer accuracy but reduces reasoning informativeness by 38.9% on average. Models reach correct answers through pattern-matching shortcuts rather than genuine inferential reasoning, becoming less auditable despite higher accuracy scores.
Supervised fine-tuning improves final-answer accuracy on benchmarks but cuts Information Gain by 38.9 percent, meaning models generate correct answers through post-hoc rationalization rather than genuine inferential steps. Standard metrics miss this degradation because they only measure final correctness.
Three faithfulness tests show fine-tuned models generate reasoning chains that less reliably influence final outputs. Early termination, paraphrasing, and filler substitution all produce invariant answers more often after fine-tuning, suggesting reasoning becomes performative rather than functional.
Supervised fine-tuning makes model outputs look correct—proper JSON structure, valid identifiers, expected sections—without making them physically feasible. The model learns surface features of solutions, not the reasoning to construct valid ones.
Logit lens analysis shows models trained with hidden CoT tokens compute correct answers in layers 1-3, then actively suppress these representations in final layers to produce format-compliant filler output. The reasoning is fully recoverable from lower-ranked token predictions.
Only ~20% of tokens exhibit high entropy as pivotal reasoning decision points; RLVR primarily adjusts these forking tokens. Training exclusively on them matches or exceeds full-gradient performance, revealing that the minority carries the learning signal.
Task accuracy peaks at intermediate CoT length, with optimal length increasing alongside task difficulty but decreasing with model capability. RL training naturally gravitates toward shorter chains as models improve, revealing that simplicity emerges from reward signals rather than explicit training.
Chain of Draft achieves equivalent accuracy to standard chain-of-thought on arithmetic, symbolic, and commonsense tasks while using only 7.6% of tokens. The 92.4% of removed tokens served style and documentation, not computation.
DataAlchemy experiments show CoT fails systematically under distributional shifts in task, length, and format. Models produce fluent but logically inconsistent reasoning — imitating reasoning form without valid underlying logic.