AI explanations can sound convincing without actually matching how the AI reached its answer — so why do we trust them anyway?
Why do reasoning traces improve answer acceptance without improving task performance?
This explores why showing a model's step-by-step reasoning makes people (and automated judges) more willing to trust its answer, even when the reasoning doesn't make the answer more likely to be right.
This explores why visible reasoning makes an answer more convincing without making it more correct. The corpus doesn't have a study that directly measures human acceptance rates. It does have something more useful: a cluster of findings showing that the reasoning text and the final answer are much more loosely connected than they look. If the trace isn't what produces the answer, then its persuasive effect and its effect on accuracy can come apart, and the trace ends up working like a performance.
The strongest evidence comes from deliberately breaking the reasoning. Models trained on irrelevant or corrupted traces solve problems about as well as models trained on correct ones, and sometimes generalize better to unfamiliar problems Do reasoning traces need to be semantically correct?. Invalid traces often lead to correct answers, and the tokens in a 'thinking' section are generated exactly like any other output. The trace matches the answer because the model learned the format, not because it ran the logic Do reasoning traces actually cause correct answers?. Prompting research points the same way: invalid chain-of-thought examples work about as well as valid ones, and formatting and layout matter far more than whether the logic holds What makes chain-of-thought reasoning actually work?. So the thing a reader inspects to decide whether to trust an answer is mostly not where the answer comes from.
This explains the acceptance gap. People read step-by-step prose the way they read a colleague's working, as evidence of a process they can check. The anthropomorphism finding names this as a safety risk: traces look like reasoning, so we credit them as reasoning Do reasoning traces actually cause correct answers?. Training methods can make this worse. Some methods keep any self-generated reasoning that ends in a correct answer Can models improve by filtering only on answer correctness?. Others reward reasoning by how strongly it predicts the reference answer Can reasoning improvement work without answer verification?. Both work, but neither checks whether the steps themselves are sound. They select for traces that lead to the right answer, which is not the same as traces that are true.
The traces aren't pure decoration, though, and this nuance is worth knowing. A few sentences really do steer what comes next. Planning and backtracking moments act as pivots that change later output Which sentences actually steer a reasoning trace?. Much of the rest is filler, or even harmful. Accuracy peaks at a middle length and then drops Why does chain of thought accuracy eventually decline with length?. Reasoning that keeps going after the answer is already settled makes models worse learners Does every correct chain-of-thought trace improve fine-tuning?. Reasoning models also tend to wander and drop promising paths too early Why do reasoning models abandon promising solution paths?. For simple questions, adding steps can hurt compared with answering directly Why do some questions perform better without step-by-step reasoning?. Longer, more elaborate traces look more thorough and earn more trust, yet past a point they often mean worse performance.
The twist: the same weakness applies when AI judges AI. Reward models that write out reasoning before they score raise the ceiling on how well they evaluate Can reward models benefit from reasoning before scoring?. That puts a judge that reasons in charge of grading a model that reasons, and both are open to well-formatted text that doesn't hold up. A more reliable signal than how good a trace looks is how confident the model is at each step. Checking step by step catches breakdowns that a polished overall trace hides Does step-level confidence outperform global averaging for trace filtering?. If you want to know whether to trust a reasoning trace, look at where it hesitates, not at how fluent it sounds.
Sources 12 notes
Models trained on systematically irrelevant traces maintain solution accuracy and sometimes improve out-of-distribution generalization, suggesting traces function as computational scaffolding rather than meaningful reasoning steps.
R1's intermediate tokens carry no special execution semantics and are generated identically to other LLM output. Invalid traces frequently produce correct answers, proving traces are not causally necessary—they correlate with answers via learned formatting, not functional reasoning.
Research shows training format shapes reasoning strategy 7.5× more than domain, demo position swings accuracy 20%, and invalid CoT prompts work as well as valid ones. CoT is pattern-guided generation, not formal logic.
STaR demonstrates that self-generated rationales filtered exclusively by answer correctness improve reasoning performance significantly. On CommonsenseQA, this correctness-filtered approach achieved 72.5% accuracy, outperforming direct answer fine-tuning and closing the gap with models 30 times larger.
VeriFree bypasses answer verification entirely by using the conditional probability of reference answers given generated reasoning traces as both reward signal and training weight. This approach matches or surpasses verifier-based methods on MMLU-Pro, GPQA, and SuperGPQA without rule-based or model-based verifiers.
Show all 12 sources
Counterfactual resampling, attention analysis, and causal suppression all identify planning and backtracking sentences as thought anchors—sparse critical points that guide subsequent reasoning. These are functional pivots, not noise.
Task accuracy peaks at intermediate CoT length, with optimal length increasing alongside task difficulty but decreasing with model capability. RL training naturally gravitates toward shorter chains as models improve, revealing that simplicity emerges from reward signals rather than explicit training.
Post-conclusion reasoning—where the model keeps exploring after sufficient evidence for the answer—degrades supervised fine-tuning despite preserving correctness. Removing only this tail improves learning more than removing equally-long random suffixes, proving the harm comes from unnecessary exploration, not length.
Reasoning LLMs exhibit two reinforcing failures: wandering (invalid exploration) and underthinking (premature path-switching). Decoding-level interventions like thought-switching penalties improve accuracy without fine-tuning, suggesting viable solutions exist but are abandoned prematurely.
Saliency analysis reveals that CoT prompting fails when question information doesn't aggregate into the prompt structure before reasoning begins. For simple questions, direct question-to-answer flow outperforms step-by-step reasoning, showing the optimal prompt depends on question type, not just task category.
Three independent teams (RRM, RM-R1, DeepSeek-GRM) discovered that adding chain-of-thought reasoning before reward scoring enables adaptive test-time compute scaling for evaluation. Reasoning-based approaches raise the capability ceiling of reward models beyond what outcome-based evaluation achieves.
Local step-level confidence catches reasoning breakdowns that global averaging masks and enables early stopping before traces complete. This approach achieves comparable accuracy gains to naive majority voting with far fewer generated traces, proving trace quality matters more than quantity.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Local Coherence or Global Validity? Investigating RLVR Traces in Math Domains
- Explaining Too Much? Understanding How Large Language Model Reasoning Traces Influence Performance and Metacognition
- Beyond Semantics: The Unreasonable Effectiveness of Reasonless Intermediate Tokens
- The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexity
- Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces!
- Diagnosing Harmful Continuation in Answer-Correct Long-CoT Training Traces
- Understanding and Mitigating Premature Confidence for Better LLM Reasoning
- What Characterizes Effective Reasoning? Revisiting Length, Review, and Structure of CoT