Why do language models produce verbose reasoning when asked to think step by step?
This explores why step-by-step prompting produces so much text — and the corpus suggests most of that text is style and format compliance, not the actual computation.
This reads the question as: when a model 'shows its work,' how much of that work is real reasoning versus performance? The corpus is surprisingly blunt — most of the verbosity is decoration. Chain of Draft showed that you can match standard chain-of-thought accuracy on arithmetic, symbolic, and commonsense tasks while spending only 7.6% of the tokens; the other 92.4% served documentation and style, not the computation that produced the answer Can minimal reasoning chains match full explanations?. So the first answer to 'why so verbose' is: because the verbosity was never load-bearing in the first place.
Where does the habit come from? It looks like a training artifact rather than a thinking requirement. Models trained to reason in hidden tokens actually compute the correct answer in their earliest layers, then actively suppress that representation to emit format-compliant filler — the real answer is recoverable from lower-ranked predictions while the visible output performs being-a-reasoner Do transformers hide reasoning before producing filler tokens?. Other architectures push this further: depth-recurrent models, Heima, and Coconut scale test-time compute entirely in latent space, with no verbalized steps at all, which suggests the spoken-out chain is a learned convention, not the mechanism Can models reason without generating visible thinking tokens?.
That convention is mimicry of a form. Reasoning traces behave as persuasive appearances rather than faithful records — invalid logical steps perform nearly as well as valid ones, and corrupted traces generalize comparably Do reasoning traces show how models actually think?. Chain-of-thought works largely by constraining the model to reproduce familiar reasoning *shapes* from training, and it degrades predictably under distribution shift, which is the signature of imitation rather than genuine inference Does chain-of-thought reasoning reveal genuine inference or pattern matching?. And the steps often don't even cause the answer: chains routinely fail both causal sufficiency and causal necessity Do language models actually use their reasoning steps?.
Here's the twist that makes this worth knowing: the verbosity isn't *always* empty — it's empty mostly on easy problems. Activation probes show models commit to an answer internally long before they finish writing on easy tasks (pure performance), but on genuinely hard tasks the unfolding text tracks real belief updates with detectable inflection points. Probe-guided early exit can cut up to 80% of tokens with no accuracy loss precisely because so much of the chain on easy items is ceremony Does chain-of-thought reasoning reflect genuine thinking or performance?. The model can't tell when it's padding, so it pads uniformly.
The practical upshot is that 'when to be verbose' is a learnable routing decision, not an inherent property. Thinkless trains a single model to choose between extended reasoning and a direct answer, self-calibrating without difficulty labels Can models learn when to think versus respond quickly?. So the honest answer to the question: models are verbose because they were rewarded for producing the visible form of reasoning, that form is mostly stylistic, and only recently are we teaching them to spend words in proportion to how much thinking the problem actually needs.
Sources 8 notes
Chain of Draft achieves equivalent accuracy to standard chain-of-thought on arithmetic, symbolic, and commonsense tasks while using only 7.6% of tokens. The 92.4% of removed tokens served style and documentation, not computation.
Logit lens analysis shows models trained with hidden CoT tokens compute correct answers in layers 1-3, then actively suppress these representations in final layers to produce format-compliant filler output. The reasoning is fully recoverable from lower-ranked token predictions.
Multiple architectures—depth-recurrent models, Heima, and Coconut—demonstrate that test-time compute scales through hidden state iteration rather than token generation. This suggests verbalization is a training artifact, not a reasoning requirement.
LLM reasoning traces perform as persuasive appearances rather than reliable explanations of computation. Invalid logical steps perform nearly as well as valid ones, and corrupted traces generalize comparably, showing that semantic correctness is not what produces the performance gains.
CoT works by constraining models to reproduce familiar reasoning patterns from training, not by enabling novel symbolic reasoning. Performance degrades predictably under distribution shifts—the signature of imitation rather than capability emergence.
LLM reasoning chains fail both causal sufficiency (steps don't always matter) and causal necessity (spurious steps are common). Research shows most CoT evaluation measures output quality, not whether reasoning actually caused the answer.
Activation probes show models commit to answers internally long before finishing their reasoning on easy tasks, but on hard tasks the reasoning process tracks real belief updates with detectable inflection points. Probe-guided early exit reduces tokens by up to 80 percent without accuracy loss.
Thinkless trains a single model to select between extended reasoning and direct responses using DeGRPO, which decouples mode selection from answer refinement. This prevents mode collapse and enables self-calibrated routing without explicit difficulty labels.