How does latent state recursion differ mechanistically from chain-of-thought prompting?
This explores the mechanical difference between two ways a model can 'think': recursing on an internal latent state (reasoning that never becomes words) versus chain-of-thought, where the model writes out intermediate steps as text and reads them back.
This explores the mechanical difference between two ways a model can 'think': recursing on an internal latent state (reasoning that never becomes words) versus chain-of-thought (CoT), where the model writes intermediate steps as tokens and conditions on them. The cleanest way to see the gap is that CoT routes reasoning *through the token stream* — each step is generated text the model then reads back — while latent recursion keeps the work in continuous internal vectors that never surface as language. One striking piece of evidence that these are genuinely separate machinery: you can trigger reasoning by directly steering a single internal feature, with no step-by-step prompt at all, and it matches or beats CoT across six model families — and it activates *early* in generation, overriding surface instructions Can we trigger reasoning without explicit chain-of-thought prompts?. That suggests latent reasoning is a built-in capability the text scaffold of CoT merely coaxes out, not creates.
The mechanistic catch is that the token stream isn't just a readout — it's load-bearing. CoT gets exact likelihoods, sampling, and KV-cache reuse for free *because* each step is a discrete token with a probability. Continuous latent thoughts lose all of that, which is why work like NF-CoT has to bolt a normalizing flow onto the causal stream just to recover tractable likelihoods and trajectory scoring for non-verbal reasoning Can continuous thoughts have tractable likelihoods for sampling and scoring?. So the difference isn't only 'words vs. vectors' — it's that discretizing into tokens hands you a free supervision and scoring substrate that latent recursion has to reconstruct by hand.
That reconstruction is exactly where latent reasoning tends to break. Train it on outcome reward alone and you get a dual collapse: gradients attenuate as they flow back along the latent steps, and the latent space drifts without any semantic anchor. Robust latent reasoning needs *dense* per-step supervision plus explicit pressure to preserve the geometry of the space rather than compressing it away Why does latent chain-of-thought fail so easily in training?. CoT sidesteps this because its 'supervision' is baked into every generated token — the chain is its own running record, which is also why local memorization on the immediately preceding tokens drives up to 67% of CoT's reasoning errors Where do memorization errors arise in chain-of-thought reasoning?. Each mechanism carries its own failure signature: latent recursion drifts and starves; CoT clings too hard to nearby tokens.
Here's the part you might not expect: a lot of what looks like 'reasoning' in CoT turns out to be the *form* of the text doing the work, not internal computation. Invalid CoT prompts work nearly as well as valid ones, training format shapes the strategy 7.5× more than the domain, and demo position can swing accuracy 20% — CoT is pattern-guided generation, not formal logic What makes chain-of-thought reasoning actually work?. It's constrained imitation of reasoning's shape, and it degrades predictably under distribution shift — the fingerprint of imitation rather than genuine inference Does chain-of-thought reasoning reveal genuine inference or pattern matching?. If much of CoT's lift comes from its textual scaffold rather than the thinking, that reframes latent recursion's pitch: it's an attempt to keep the computation while discarding the verbose token theater — and Chain of Draft hints the theater is most of the cost, matching full CoT accuracy at 7.6% of the tokens, with the other 92% serving style and documentation, not computation Can minimal reasoning chains match full explanations?.
The through-line: CoT and latent recursion differ in *where the reasoning lives and what that location buys you*. Tokens give CoT a free, interpretable, scorable record — at the price of verbosity, local-token memorization, and form-over-substance imitation. Latent recursion promises denser, cheaper, non-verbal computation — at the price of vanishing gradients, drifting representations, and lost tractability you have to engineer back. One reasons out loud and pays for the words; the other reasons in the dark and pays to build the lights.
Sources 7 notes
SAE-identified reasoning features can be directly steered to match or exceed chain-of-thought performance across six model families. This reasoning mode activates early in generation and overrides surface-level instructions, suggesting latent reasoning is a fundamental capability independent of explicit prompting.
NF-CoT models continuous thoughts as an autoregressive normalizing flow inside the LLM's causal stream, recovering exact likelihood, probabilistic sampling, and KV-cache compatibility. This enables policy-gradient refinement and trajectory scoring on non-verbal reasoning, matching the tractability of textual CoT.
Outcome supervision alone causes gradient attenuation along latent steps and lets the latent space wander without semantic grounding. Robust latent reasoning requires both dense trajectory supervision and space supervision that preserves geometric structure rather than compressing it.
STIM framework identifies local, mid-range, and long-range memorization sources in CoT reasoning. Local memorization—based on preceding tokens—accounts for up to 67% of reasoning errors, especially as complexity increases and distributional shift occurs.
Research shows training format shapes reasoning strategy 7.5× more than domain, demo position swings accuracy 20%, and invalid CoT prompts work as well as valid ones. CoT is pattern-guided generation, not formal logic.
CoT works by constraining models to reproduce familiar reasoning patterns from training, not by enabling novel symbolic reasoning. Performance degrades predictably under distribution shifts—the signature of imitation rather than capability emergence.
Chain of Draft achieves equivalent accuracy to standard chain-of-thought on arithmetic, symbolic, and commonsense tasks while using only 7.6% of tokens. The 92.4% of removed tokens served style and documentation, not computation.