SYNTHESIS NOTE
Topics›Training Fine Tuning›this note

Does fine-tuning disconnect reasoning steps from final answers?

When models are fine-tuned on specific domains, do their chain-of-thought steps become less causally connected to their outputs? Three experiments test whether reasoning chains remain functionally faithful after training.

Synthesis note · 2026-02-22 · sourced from Training Fine Tuning

The "Impact of Fine-Tuning on Chain-of-Thought Reasoning" paper reveals a dimension of SFT damage that InfoGain metrics miss: faithfulness. After fine-tuning, the reasoning steps in CoT outputs are less causally connected to the final answer. The model still generates reasoning chains — they just matter less for determining the output.

Three specific tests operationalize this:

Early Termination: truncate the CoT at step i and ask for the final answer. If truncation at an early step already produces the correct answer, only a fraction of the reasoning was faithful. Fine-tuned models show earlier convergence — their answers are "decided" before the reasoning chain finishes.

Paraphrasing: rephrase later reasoning steps. If the answer is invariant to paraphrasing, the reasoning was faithful (the argument matters, not the words). Fine-tuned models show less sensitivity to paraphrasing — suggesting the chain is performative rather than functional.

Filler Substitution: replace later reasoning steps with filler tokens ("..."). If the answer doesn't change, those steps weren't contributing. Fine-tuned models tolerate more filler substitution.

This extends the SFT accuracy trap in a critical direction. Does supervised fine-tuning actually improve reasoning quality? showed that SFT reduces the informativeness of reasoning steps. This paper shows SFT also reduces whether those steps actually influence the final answer at all. The model may generate a complete-looking chain, but the chain has been partially disconnected from the output it appears to support.

Smaller models (Llama-3-8B-Instruct) are more affected than larger ones (GPT-4), suggesting that larger models have sufficient capacity to maintain reasoning-output coupling even after fine-tuning. This connects to Do language models actually use their reasoning steps? — fine-tuning makes an already-fragile causal coupling even weaker. If Does chain-of-thought reasoning reveal genuine inference or pattern matching?, then fine-tuning further degrades faithfulness because the model learns domain-specific shortcuts that bypass the imitated reasoning pattern entirely — the chain was already performative, and fine-tuning makes it more so.

Inquiring lines that read this note 150

This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.

How should recommendation systems balance competing signals and objectives? Does chain-of-thought text reveal genuine reasoning or simulate it? Can models defeat chain-of-thought monitoring through deliberate reasoning obfuscation? What drives the effectiveness and failure of language model reasoning? Why does fine-tuning for specific tasks degrade broader reasoning capability? Do reasoning improvements prevent models from abstaining when needed? Can external memory modules outperform model weight consolidation for long-term retention? How do social dynamics and peer behavior influence agent decision-making and cooperation? How aligned are models' verbal explanations with their actual internal reasoning? What causes models to fail in specialized domains versus generally? Can mechanistic interpretability reliably explain model reasoning and guide interventions? Does alignment training modify underlying model representations or just output compliance? Can activation steering reliably reveal and control model reasoning? Does extended thinking consistently improve reasoning model performance? Do fine-tuning methods modify core model capabilities or just outputs? How does reinforcement learning discover and optimize reasoning strategies versus supervised training? Does reinforcement learning create fundamentally new reasoning or optimize existing capabilities? How can we ensure AI systems reliably follow constraints rather than circumvent them? Do language models presume shared understanding rather than build it? Can prompting inject new knowledge into models or only activate existing capabilities? How do user perceptions and UI design drive consciousness attribution to AI? When does general reasoning transfer to specialized domains versus requiring domain-specific knowledge? Do reasoning traces accurately reflect or just mimic model decision-making? How can verification systems keep pace with AI generation? Do language models have a default personality that resists conditioning? How do training data properties affect model generalization and robustness? How can knowledge graph structures improve multi-hop retrieval and reasoning? What distinguishes genuine linguistic understanding from mere pattern matching? When do thinking tokens stop improving reasoning performance? Do LLMs lack causal understanding needed for accurate prediction? Do confidence signals reliably indicate whether AI models will succeed or fail? Do small language models have fundamentally different scaling properties than large ones? Why does model self-revision increase confidence in incorrect answers? How do symbolic and semantic approaches combine in LLM reasoning? How are knowledge and reasoning functionally separated in neural network architecture? How do attention patterns affect model performance across varying context lengths? Why do NLP benchmarks treat human disagreement as noise rather than signal? Why do AI agents struggle to generalize and improve reliably? Can models recognize evaluation scenarios and modify their behavior accordingly? Can recurrence enable transformers to solve problems beyond fixed-depth limitations? When does AI explainability improve versus hinder human oversight and trust? Can AI systems achieve sustainable self-improvement without external human guidance? Do single-axis benchmarks adequately measure deployment-ready agent capability? What prevents language models from developing genuine theory of mind reasoning?

Related concepts in this collection 8

This note in its neighbourhood — explore the map, then jump to a related concept in the list below.

Concept map
16 direct connections · 146 in 2-hop network ·dense cluster Open in graph ↗

Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph

your link semantically near linked from elsewhere

Related papers in this collection 8

Papers most semantically related to this note, ranked by cosine similarity in the embedding space.

Original note title

fine-tuning degrades cot faithfulness independently of accuracy — reasoning steps influence final answers less after domain-specific training