SYNTHESIS NOTE
Does supervised fine-tuning improve reasoning or just answers?
Explores whether training models on question-answer pairs actually strengthens their reasoning quality or merely optimizes them toward correct outputs through shortcuts. This matters for deploying AI in domains like medicine where reasoning must be auditable.
Post angle for Medium / LinkedIn
Hook: "Every AI benchmark measures accuracy. What if accuracy is exactly the wrong thing to measure when deploying AI in high-stakes domains?"
The finding: The Knowledge or Reasoning paper introduces two new metrics — Knowledge Index (KI: factual correctness of each reasoning step) and Information Gain (InfoGain: how much each reasoning step reduces uncertainty toward the final answer). When they apply these metrics to SFT-trained models on medical and mathematical tasks, they find that SFT raises final-answer accuracy while cutting InfoGain by 38.9%. Models get more answers right while reasoning toward them less informationally.
The mechanism: SFT rewards answers, not reasoning paths. Training data has question-answer pairs. The loss function anchors on the correct final output. Models learn the most efficient path to the right answer in the training distribution — often domain-specific shortcuts, pattern matches, and frequency-weighted heuristics that produce the correct answer without the inferential chain that would justify it. The reasoning in the output becomes post-hoc rationalization.
Why this matters for deployment: High-stakes domains don't just need correct answers — they need auditable reasoning. Medical decision support must show clinical logic. Legal AI must demonstrate how conclusions follow from statute and precedent. Financial AI must show how recommendations connect to market data and regulatory context. SFT improves the answer, but may make the reasoning path less meaningful — more verbose decoration around the correct output than the pathway that produced it.
The measurement problem: Standard benchmarks measure what's easy to measure: whether the final answer matches the ground truth. InfoGain and KI require decomposing reasoning chains and evaluating each step against external ground truth — expensive and difficult to automate at scale. So the measurement gap persists, and every organization that deploys based on benchmark accuracy is systematically blind to the reasoning quality regression.
The connection: This extends the existing cluster of overthinking findings into the training dimension. Does extended thinking actually improve reasoning or just increase variance? at inference-time. Does reasoning fine-tuning make models worse at declining to answer? at training-time for a different cost (calibration). The SFT accuracy trap is the third entry: training-time cost to reasoning quality.
Platform: Medium (1000–1400 words). Could lead with the FALM / medical AI deployment angle, then introduce the measurement framework.
Inquiring lines that read this note 131
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
Does AI assistance enhance or undermine professional skill development?- Can debugging skills be validated if AI training degraded them first?
- Can personalized AI learning systems actually widen rather than narrow educational gaps?
- When randomized trials miss users most likely to benefit from AI tools?
- Does AI tutoring improve exam performance without reducing independent capability?
- What design features make tutoring AI preserve learning better than answer-giving AI?
- How does AI substitute polished style for actual expert judgment?
- Can organized response format trick users into overestimating AI reliability?
- Does polished presentation actually substitute for expert judgment in AI outputs?
- Does evaluating AI output require different cognitive skills than solving problems directly?
- Can AI evaluation tools solve the verification problem they help create?
- What role could knowledge custodians play in validating AI output?
- Does verification-conditioned use concentrate AI in tasks where juniors already have expertise?
- How should we redesign benchmarks to catch conservative bias in reasoning tasks?
- Why do benchmark scores rise while reasoning quality declines?
- What training methods make models more persuasive but less factually accurate?
- Can preference optimization training make models worse at detecting false presuppositions?
- What alternatives to RLHF better preserve truth-seeking in AI outputs?
- Does RLHF produce different tacit knowledge than pretraining or constitutional AI methods?
- Could AI assessment quality differ across subjects or question formats?
- What makes reasoning auditable in medical AI decision support?
- Does optimizing for differential diagnosis accuracy risk pushing AI systems toward premature problem-solving?
- How does expert annotation instability affect medical AI benchmarking?
- Does medical AI accuracy depend more on knowledge or reasoning ability?
- What prospective trials are needed to validate AI diagnostic claims?
- Can clinicians reliably distinguish high-quality AI advice from low-quality advice by appearance alone?
- Can annotation and explanation labels reduce automation bias in clinical settings?
- Does AI change clinician cognition or just increase reliance on predictions?
- Does low-stakes experimental AI performance predict real-world clinical or financial decision outcomes?
- Why do single examples trigger large reasoning improvements in models?
- Can training improve reasoning coherence without improving actual correctness?
- Can a single correct example seed exponential improvement in mathematical reasoning?
- Can reasoning catalyst data serve as a stable foundation for test-time training?
- How can one training example improve reasoning across thousands of unseen problems?
- Can attribute decomposition improve other interactive reasoning tasks beyond clinical questioning?
- Why does reasoning training improve math but hurt knowledge tasks?
- Why does eliminating proxy-model filtering improve reasoning emergence in pretraining?
- Can models maintain reasoning-output coupling while improving domain accuracy?
- Can smaller amounts of diverse reasoning demonstrations replace exhaustive factual training data?
- Can small demonstration sets unlock general reasoning without large question data?
- How does question difficulty and breadth affect what models learn to reason?
- Which domains need knowledge injection versus reasoning-focused training?
- How do procedural versus factual knowledge differ in pretraining versus fine-tuning?
- How much of Occamy's result comes from training versus the base model?
- Which medical tasks benefit most from domain-adaptive pretraining?
- How does optimizing for accuracy during training degrade downstream reasoning quality?
- Does training for better reasoning reduce an AI system's ability to abstain?
- How does critique fine-tuning on one problem unlock broader reasoning?
- Why does reasoning fine-tuning reduce model abstention capacity by 24 percent?
- Does domain training degrade reasoning ability even when benchmark scores rise?
- Why does fine-tuning degrade reasoning quality even as accuracy improves?
- Why does fine-tuning sometimes damage chain-of-thought reasoning even when accuracy improves?
- Why does domain accuracy improve while reasoning quality degrades after supervised fine-tuning?
- Can fine-tuning ever teach semantic inference instead of amplifying training shortcuts?
- Does supervised fine-tuning improve accuracy while damaging the quality of reasoning?
- Does fine-tuning models for specific tasks destroy their ability to reason?
- Does reasoning fine-tuning actually reduce a model's ability to abstain?
- How does fine-tuning on natural language inference affect fallacy susceptibility?
- Do models trained for reasoning lose their ability to decline questions?
- What happens when reasoning fine-tuning eliminates model refusal mechanisms entirely?
- Why does supervised fine-tuning degrade reasoning quality despite raising accuracy?
- Why does SFT reduce reasoning quality even when improving domain accuracy?
- Does reasoning fine-tuning actually damage a model's ability to abstain?
- Does SFT degrade reasoning quality while improving domain accuracy?
- What training cost tradeoffs exist between fine-tuning and other knowledge injection methods?
- How does data quality mismatch create reasoning degradation in supervised fine-tuning?
- Does reasoning fine-tuning actually harm a model's ability to abstain?
- Can reasoning fine-tuning improve both capability and instruction compliance together?
- Why does reasoning fine-tuning reduce a model's ability to abstain?
- Can models be trained to explain instead of imitate answers?
- Does supervised fine-tuning improve reasoning or just response formatting?
- Does fine-tuning push models toward reasoning shortcuts that bypass the chain entirely?
- Why does reasoning fine-tuning suppress the confidence signals that adaptive retrieval needs?
- How does supervised fine-tuning degrade chain-of-thought faithfulness over time?
- Why does reasoning fine-tuning reduce models' ability to abstain?
- How does preference learning differ from supervised finetuning for reasoning?
- Can reasoning improvements be attributed when optimizer and scaffold are unknown?
- Does fine-tuning improve domain accuracy at the cost of reasoning quality?
- Does looped pretraining build reasoning more efficiently than supervised fine-tuning?
- Do tokens beyond a critical threshold actually improve reasoning quality?
- Can we improve reasoning by amplifying information at mutual information peaks?
- Why does increased reasoning effort fail to improve accuracy consistently?
- Does training on critiques of noisy responses produce deeper understanding than imitating correct ones?
- How do contrasting examples improve AI feedback quality over generic suggestions?
- Why do explicit quality criteria outperform learning quality from examples alone?
- What makes some training data teach brittle answers versus robust reasoning?
- Can verifier-guided search catch factual errors that reasoning training cannot?
- What makes the verifier the load-bearing component of reasoning training?
- What makes training data quality more important than quantity for reasoning?
- Why does training data format shape reasoning strategy more than content?
- Can question quality be trained separately from the decision to ask?
- What training approach enables models to proactively request clarification?
- Why does document perplexity stay low while question-answering accuracy drops?
- Does formal reasoning training actively degrade social reasoning ability?
- Why does increasing reasoning not improve AI social reasoning performance?
- Can AI evaluation match human judgment quality in structured domain tasks?
- Can AI explainability help humans and machines reach better joint decisions?
- Does reinforcement learning preserve reasoning quality better than supervised fine-tuning?
- Can reinforcement learning fix the reasoning gaps that supervised fine-tuning misses?
- Can single-problem fine-tuning match full RL pipeline reasoning gains?
- Can approximate or noisy reference answers work for RL-based reasoning training?
- Why does automated evaluation consistently overestimate research quality?
- Does distilling experiment outcomes into reusable insights improve hypothesis quality over ranking alone?
- Why do AI benchmarks measure accuracy instead of reasoning quality?
- Could deliberate training toward benchmarks overstate real gains in AI capability?
- Can adversarial critics force genuine reasoning the same way critique fine-tuning does?
- Can evaluation trajectories and interaction histories replace single-answer scoring?
- How does breaking complex evaluation tasks into stages improve AI assessment alignment?
Related concepts in this collection 10
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Concept map
Does supervised fine-tuning improve reasoning or…
Does supervised fine-tuning actually improve reaso…
Does reasoning fine-tuning make models worse at de…
Does extended thinking actually improve reasoning …
Does critiquing errors teach deeper understanding …
Why do better reasoning models ignore instructions…
Can language models solve ToM benchmarks without r…
Why does SFT-then-RL training follow a predictable…
Does preference optimization harm conversational u…
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
your link
semantically near
linked from elsewhere
-
Does supervised fine-tuning actually improve reasoning quality?
While SFT boosts final-answer accuracy, does it degrade the quality and informativeness of the reasoning steps that justify those answers? This matters for high-stakes domains requiring auditable decision-making.
the underlying insight this post dramatizes
-
Does reasoning fine-tuning make models worse at declining to answer?
When models are trained to reason better, do they lose the ability to say 'I don't know'? This matters for high-stakes applications like medical and legal AI that depend on appropriate uncertainty.
parallel SFT cost: calibration vs. reasoning quality
-
Does extended thinking actually improve reasoning or just increase variance?
When models think longer, do they reason better, or do they simply sample from a wider distribution of outputs that happens to cover correct answers more often? This matters because it determines whether test-time compute is genuinely scaling reasoning capability.
inference-time version of the same accuracy vs. quality trade-off
-
Does critiquing errors teach deeper understanding than imitating correct answers?
Can training models to critique flawed responses build better structural understanding than standard supervised fine-tuning on correct answers? This matters because it reveals whether deep reasoning requires engaging with failure modes rather than pattern matching.
counter-strategy: CFT addresses the SFT accuracy trap by replacing correct-answer imitation with structured failure analysis as the training objective
-
Why do better reasoning models ignore instructions?
As models develop stronger reasoning abilities through training, they appear to become worse at following specified constraints. Is this an unavoidable trade-off, and what causes it?
a third dimension of SFT/RL training cost: SFT degrades reasoning quality (this note), reasoning training degrades instruction adherence (instruction-following deficit), and both reflect the same pattern — optimizing one capability structurally degrades another
-
Can language models solve ToM benchmarks without real reasoning?
Do current theory-of-mind benchmarks actually measure mental state reasoning, or can models exploit surface patterns and distribution biases to achieve high scores? This matters because it determines whether benchmark performance indicates genuine understanding.
ToM benchmarks are a concrete case of the SFT accuracy trap: SFT achieves competitive ToM scores without reasoning training, suggesting benchmarks reward structural pattern exploitation rather than genuine mental state reasoning
-
Why does SFT-then-RL training follow a predictable three-phase pattern?
When expert data diverges from a model's learned patterns, SFT-then-RL training exhibits disruption, readaptation, and overfitting phases. Understanding this progression could improve how we combine imitation and reinforcement learning.
temporal dynamics: CHORD reveals the SFT accuracy trap as the first phase of a three-phase progression; RL can recover from SFT's reasoning degradation but only if SFT and RL are integrated as a continuous spectrum rather than hard-sequenced stages
-
Does preference optimization harm conversational understanding?
Exploring whether RLHF training that rewards confident, complete responses undermines the grounding acts—clarifications, checks, acknowledgments—that actually build shared understanding in dialogue.
parallel training-induced degradation: SFT degrades reasoning quality (this note) while RLHF degrades conversational grounding; both demonstrate that optimizing for what benchmarks and raters measure structurally erodes capabilities that require different evaluation frameworks
-
Can identical outputs hide broken internal representations?
Can neural networks produce correct outputs while having fundamentally fractured internal structure that prevents generalization and creativity? This challenges our assumptions about what performance benchmarks actually measure.
FER provides the representational diagnosis: SFT may produce fractured internal representations that yield correct answers through pattern-matching shortcuts while the underlying structure is broken in ways standard benchmarks cannot detect
-
Does fine-tuning disconnect reasoning steps from final answers?
When models are fine-tuned on specific domains, do their chain-of-thought steps become less causally connected to their outputs? Three experiments test whether reasoning chains remain functionally faithful after training.
a second dimension of SFT damage beyond InfoGain: fine-tuning reduces how much reasoning steps causally influence the final answer, making the chain performative rather than functional; together with InfoGain degradation, SFT damages both reasoning quality and reasoning faithfulness
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces!
- Learning to Reason for Factuality
- Knowledge or Reasoning? A Close Look at How LLMs Think Across Domains
- Echo Chamber: RL Post-training Amplifies Behaviors Learned in Pretraining
- A Survey on Test-Time Scaling in Large Language Models: What, How, Where, and How Well?
- Eliciting Reasoning in Language Models with Cognitive Tools
- An Enigma of Artificial Reason: Investigating the Production-Evaluation Gap in Large Reasoning Models
- Post-Completion Learning for Language Models
Original note title
the sft accuracy trap — training raises benchmark scores while degrading reasoning quality