Why do larger reasoning models show cyclicity only in later layers?
This explores what the corpus knows about 'cyclicity' in reasoning models' hidden states — loops where the model revisits a representation — and whether it can explain why such loops concentrate in later layers of bigger models.
This reads the question as being about cyclicity in the geometry of a model's hidden states — moments where the reasoning trajectory loops back on itself rather than moving straight ahead. Up front, an honesty flag: the collection has exactly one note that studies this directly, and it doesn't isolate the layer-by-layer-by-size pattern your question names. But it gives a frame that makes the pattern unsurprising, and the rest of the corpus sharpens it.
The anchor is Do reasoning cycles in hidden states reveal aha moments?, which finds distilled reasoning models run about five hidden-state cycles per sample where base models run nearly zero — and that cyclicity tracks accuracy. Crucially, these cycles map onto documented 'aha moments': the points where a model reconsiders an intermediate answer. So cyclicity isn't noise, it's the geometric fingerprint of reconsideration. That reframes your question: later-layer cycling would mean reconsideration happens where the model holds its most abstract, high-level commitments, not where it's still resolving tokens and surface form. Bigger models simply have more depth over which to push that abstract decision-making downstream.
The latent-reasoning work makes this concrete. Can models reason without generating visible thinking tokens? shows depth-recurrent architectures (Coconut, Heima) scale extra 'thinking' by iterating hidden states rather than emitting tokens — verbalization turns out to be a training artifact, not a requirement for reasoning. If reasoning is hidden-state iteration, then a cycle is literally what that iteration looks like, and you'd expect it to live in the layers that carry the semantic content worth iterating on — the later ones.
A lateral angle worth pulling: capability changes where reasoning effort concentrates rather than just how much there is. Why does chain of thought accuracy eventually decline with length? finds stronger models gravitate toward shorter chains — they spend reconsideration more selectively. Read alongside the cyclicity note, that suggests larger models aren't cycling everywhere indiscriminately; they reserve the loop for the layers where a genuine reconsideration pays off. And the failure-mode notes — Why do reasoning models abandon promising solution paths? and Do reasoning models switch between ideas too frequently? — describe the pathological version: cycling that becomes thrashing between ideas, which decoding penalties on thought-switching can curb. So there's a productive band of cyclicity and a destructive one.
The thing you might not have expected to learn: there's an active debate about whether any of this 'reconsideration' is real reasoning at all. Do reasoning traces need to be semantically correct? and What makes chain-of-thought reasoning actually work? argue traces work as computational scaffolding through pattern-matching, not logical inference — which would make later-layer cycles a learned structural habit rather than deliberate rethinking. The corpus genuinely splits here, and your layer-depth observation is exactly the kind of mechanistic evidence that could tip it one way.
Sources 7 notes
Distilled reasoning models show ~5 cycles per sample versus near-zero in base models, and cyclicity correlates with accuracy. These cycles in hidden-state reasoning graphs directly map to RL-trained models' documented aha moments—moments when models reconsider intermediate answers.
Multiple architectures—depth-recurrent models, Heima, and Coconut—demonstrate that test-time compute scales through hidden state iteration rather than token generation. This suggests verbalization is a training artifact, not a reasoning requirement.
Task accuracy peaks at intermediate CoT length, with optimal length increasing alongside task difficulty but decreasing with model capability. RL training naturally gravitates toward shorter chains as models improve, revealing that simplicity emerges from reward signals rather than explicit training.
Reasoning LLMs exhibit two reinforcing failures: wandering (invalid exploration) and underthinking (premature path-switching). Decoding-level interventions like thought-switching penalties improve accuracy without fine-tuning, suggesting viable solutions exist but are abandoned prematurely.
o1-like models frequently abandon reasoning paths mid-exploration, wasting tokens on incomplete approaches. A decoding-only penalty on thought-transition tokens (TIP strategy) discourages switching, improving accuracy on challenging math without model fine-tuning.
Models trained on systematically irrelevant traces maintain solution accuracy and sometimes improve out-of-distribution generalization, suggesting traces function as computational scaffolding rather than meaningful reasoning steps.
CoT systems reproduce the form of reasoning through pattern matching rather than performing genuine logical inference. This explains why format effects dominate content, why structurally invalid prompts succeed, and why stronger reasoning models become less instruction-compliant.