SYNTHESIS NOTE
Does supervised fine-tuning improve reasoning or just answers?
Explores whether training models on question-answer pairs actually strengthens their reasoning quality or merely optimizes them toward correct outputs through shortcuts. This matters for deploying AI in domains like medicine where reasoning must be auditable.
Post angle for Medium / LinkedIn
Hook: "Every AI benchmark measures accuracy. What if accuracy is exactly the wrong thing to measure when deploying AI in high-stakes domains?"
The finding: The Knowledge or Reasoning paper introduces two new metrics — Knowledge Index (KI: factual correctness of each reasoning step) and Information Gain (InfoGain: how much each reasoning step reduces uncertainty toward the final answer). When they apply these metrics to SFT-trained models on medical and mathematical tasks, they find that SFT raises final-answer accuracy while cutting InfoGain by 38.9%. Models get more answers right while reasoning toward them less informationally.
The mechanism: SFT rewards answers, not reasoning paths. Training data has question-answer pairs. The loss function anchors on the correct final output. Models learn the most efficient path to the right answer in the training distribution — often domain-specific shortcuts, pattern matches, and frequency-weighted heuristics that produce the correct answer without the inferential chain that would justify it. The reasoning in the output becomes post-hoc rationalization.
Why this matters for deployment: High-stakes domains don't just need correct answers — they need auditable reasoning. Medical decision support must show clinical logic. Legal AI must demonstrate how conclusions follow from statute and precedent. Financial AI must show how recommendations connect to market data and regulatory context. SFT improves the answer, but may make the reasoning path less meaningful — more verbose decoration around the correct output than the pathway that produced it.
The measurement problem: Standard benchmarks measure what's easy to measure: whether the final answer matches the ground truth. InfoGain and KI require decomposing reasoning chains and evaluating each step against external ground truth — expensive and difficult to automate at scale. So the measurement gap persists, and every organization that deploys based on benchmark accuracy is systematically blind to the reasoning quality regression.
The connection: This extends the existing cluster of overthinking findings into the training dimension. Does extended thinking actually improve reasoning or just increase variance? at inference-time. Does reasoning fine-tuning make models worse at declining to answer? at training-time for a different cost (calibration). The SFT accuracy trap is the third entry: training-time cost to reasoning quality.
Platform: Medium (1000–1400 words). Could lead with the FALM / medical AI deployment angle, then introduce the measurement framework.
Inquiring lines that read this note 98
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
Can AI-generated outputs constitute genuine knowledge or valid claims?- Can debugging skills be validated if AI training degraded them first?
- What role could knowledge custodians play in validating AI output?
- How does AI substitute polished style for actual expert judgment?
- Does evaluating AI output require different cognitive skills than solving problems directly?
- How should we redesign benchmarks to catch conservative bias in reasoning tasks?
- Could AI assessment quality differ across subjects or question formats?
- Why do benchmark scores rise while reasoning quality declines?
- Why do AI benchmarks measure accuracy instead of reasoning quality?
- Why does document perplexity stay low while question-answering accuracy drops?
- What training methods make models more persuasive but less factually accurate?
- Can preference optimization training make models worse at detecting false presuppositions?
- Why do single examples trigger large reasoning improvements in models?
- How does optimizing for accuracy during training degrade downstream reasoning quality?
- How does critique fine-tuning on one problem unlock broader reasoning?
- Can verifier-guided search catch factual errors that reasoning training cannot?
- Does domain training degrade reasoning ability even when benchmark scores rise?
- What makes training data quality more important than quantity for reasoning?
- Can training improve reasoning coherence without improving actual correctness?
- Can a single correct example seed exponential improvement in mathematical reasoning?
- Can reasoning catalyst data serve as a stable foundation for test-time training?
- How can one training example improve reasoning across thousands of unseen problems?
- Can models be trained to explain instead of imitate answers?
- Can attribute decomposition improve other interactive reasoning tasks beyond clinical questioning?
- Why does reasoning training improve math but hurt knowledge tasks?
- Why does eliminating proxy-model filtering improve reasoning emergence in pretraining?
- Can models maintain reasoning-output coupling while improving domain accuracy?
- Can smaller amounts of diverse reasoning demonstrations replace exhaustive factual training data?
- What makes some training data teach brittle answers versus robust reasoning?
- Can small demonstration sets unlock general reasoning without large question data?
- How does question difficulty and breadth affect what models learn to reason?
- Can reasoning improvements be attributed when optimizer and scaffold are unknown?
- Does training for better reasoning reduce an AI system's ability to abstain?
- Do models trained for reasoning lose their ability to decline questions?
- What happens when reasoning fine-tuning eliminates model refusal mechanisms entirely?
- What training approach enables models to proactively request clarification?
- Why does reasoning fine-tuning reduce model abstention capacity by 24 percent?
- Does fine-tuning models for specific tasks destroy their ability to reason?
- Does reasoning fine-tuning actually reduce a model's ability to abstain?
- Does reasoning fine-tuning actually damage a model's ability to abstain?
- Does reasoning fine-tuning actually harm a model's ability to abstain?
- Can reasoning fine-tuning improve both capability and instruction compliance together?
- Why does reasoning fine-tuning reduce a model's ability to abstain?
- Does fine-tuning push models toward reasoning shortcuts that bypass the chain entirely?
- Why does reasoning fine-tuning reduce models' ability to abstain?
- Why does fine-tuning degrade reasoning quality even as accuracy improves?
- Why does fine-tuning sometimes damage chain-of-thought reasoning even when accuracy improves?
- Why does domain accuracy improve while reasoning quality degrades after supervised fine-tuning?
- Can fine-tuning ever teach semantic inference instead of amplifying training shortcuts?
- Does supervised fine-tuning improve accuracy while damaging the quality of reasoning?
- How does fine-tuning on natural language inference affect fallacy susceptibility?
- Why does supervised fine-tuning degrade reasoning quality despite raising accuracy?
- Why does SFT reduce reasoning quality even when improving domain accuracy?
- Does SFT degrade reasoning quality while improving domain accuracy?
- How does data quality mismatch create reasoning degradation in supervised fine-tuning?
- Does supervised fine-tuning improve reasoning or just response formatting?
- Can reinforcement learning fix the reasoning gaps that supervised fine-tuning misses?
- How does supervised fine-tuning degrade chain-of-thought faithfulness over time?
- How does preference learning differ from supervised finetuning for reasoning?
- Does fine-tuning improve domain accuracy at the cost of reasoning quality?
- Can AI evaluation tools solve the verification problem they help create?
- What makes reasoning auditable in medical AI decision support?
- How do contrasting examples improve AI feedback quality over generic suggestions?
- Can evaluation trajectories and interaction histories replace single-answer scoring?
- Can attribute-specific preference optimization improve question quality in information-seeking?
- Can question quality be trained separately from the decision to ask?
- Does formal reasoning training actively degrade social reasoning ability?
- Why does increasing reasoning not improve AI social reasoning performance?
- Does reinforcement learning preserve reasoning quality better than supervised fine-tuning?
- Can approximate or noisy reference answers work for RL-based reasoning training?
- What training cost tradeoffs exist between fine-tuning and other knowledge injection methods?
- Which domains need knowledge injection versus reasoning-focused training?
- Can personalized AI learning systems actually widen rather than narrow educational gaps?
- Why does naive personalization fine-tuning destroy generalist reasoning?
Related concepts in this collection 10
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Concept map
Does supervised fine-tuning improve reasoning or…
Does supervised fine-tuning actually improve reaso…
Does reasoning fine-tuning make models worse at de…
Does extended thinking actually improve reasoning …
Does critiquing errors teach deeper understanding …
Why do better reasoning models ignore instructions…
Can language models solve ToM benchmarks without r…
Why does SFT-then-RL training follow a predictable…
Does preference optimization harm conversational u…
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
your link
semantically near
linked from elsewhere
-
Does supervised fine-tuning actually improve reasoning quality?
While SFT boosts final-answer accuracy, does it degrade the quality and informativeness of the reasoning steps that justify those answers? This matters for high-stakes domains requiring auditable decision-making.
the underlying insight this post dramatizes
-
Does reasoning fine-tuning make models worse at declining to answer?
When models are trained to reason better, do they lose the ability to say 'I don't know'? This matters for high-stakes applications like medical and legal AI that depend on appropriate uncertainty.
parallel SFT cost: calibration vs. reasoning quality
-
Does extended thinking actually improve reasoning or just increase variance?
When models think longer, do they reason better, or do they simply sample from a wider distribution of outputs that happens to cover correct answers more often? This matters because it determines whether test-time compute is genuinely scaling reasoning capability.
inference-time version of the same accuracy vs. quality trade-off
-
Does critiquing errors teach deeper understanding than imitating correct answers?
Can training models to critique flawed responses build better structural understanding than standard supervised fine-tuning on correct answers? This matters because it reveals whether deep reasoning requires engaging with failure modes rather than pattern matching.
counter-strategy: CFT addresses the SFT accuracy trap by replacing correct-answer imitation with structured failure analysis as the training objective
-
Why do better reasoning models ignore instructions?
As models develop stronger reasoning abilities through training, they appear to become worse at following specified constraints. Is this an unavoidable trade-off, and what causes it?
a third dimension of SFT/RL training cost: SFT degrades reasoning quality (this note), reasoning training degrades instruction adherence (instruction-following deficit), and both reflect the same pattern — optimizing one capability structurally degrades another
-
Can language models solve ToM benchmarks without real reasoning?
Do current theory-of-mind benchmarks actually measure mental state reasoning, or can models exploit surface patterns and distribution biases to achieve high scores? This matters because it determines whether benchmark performance indicates genuine understanding.
ToM benchmarks are a concrete case of the SFT accuracy trap: SFT achieves competitive ToM scores without reasoning training, suggesting benchmarks reward structural pattern exploitation rather than genuine mental state reasoning
-
Why does SFT-then-RL training follow a predictable three-phase pattern?
When expert data diverges from a model's learned patterns, SFT-then-RL training exhibits disruption, readaptation, and overfitting phases. Understanding this progression could improve how we combine imitation and reinforcement learning.
temporal dynamics: CHORD reveals the SFT accuracy trap as the first phase of a three-phase progression; RL can recover from SFT's reasoning degradation but only if SFT and RL are integrated as a continuous spectrum rather than hard-sequenced stages
-
Does preference optimization harm conversational understanding?
Exploring whether RLHF training that rewards confident, complete responses undermines the grounding acts—clarifications, checks, acknowledgments—that actually build shared understanding in dialogue.
parallel training-induced degradation: SFT degrades reasoning quality (this note) while RLHF degrades conversational grounding; both demonstrate that optimizing for what benchmarks and raters measure structurally erodes capabilities that require different evaluation frameworks
-
Can identical outputs hide broken internal representations?
Can neural networks produce correct outputs while having fundamentally fractured internal structure that prevents generalization and creativity? This challenges our assumptions about what performance benchmarks actually measure.
FER provides the representational diagnosis: SFT may produce fractured internal representations that yield correct answers through pattern-matching shortcuts while the underlying structure is broken in ways standard benchmarks cannot detect
-
Does fine-tuning disconnect reasoning steps from final answers?
When models are fine-tuned on specific domains, do their chain-of-thought steps become less causally connected to their outputs? Three experiments test whether reasoning chains remain functionally faithful after training.
a second dimension of SFT damage beyond InfoGain: fine-tuning reduces how much reasoning steps causally influence the final answer, making the chain performative rather than functional; together with InfoGain degradation, SFT damages both reasoning quality and reasoning faithfulness
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces!
- Learning to Reason for Factuality
- Knowledge or Reasoning? A Close Look at How LLMs Think Across Domains
- Echo Chamber: RL Post-training Amplifies Behaviors Learned in Pretraining
- A Survey on Test-Time Scaling in Large Language Models: What, How, Where, and How Well?
- Eliciting Reasoning in Language Models with Cognitive Tools
- An Enigma of Artificial Reason: Investigating the Production-Evaluation Gap in Large Reasoning Models
- Post-Completion Learning for Language Models
Original note title
the sft accuracy trap — training raises benchmark scores while degrading reasoning quality