Can recurrent transformers learn genuinely new computations beyond inference stages?
This explores whether looping a transformer's layers actually produces new kinds of computation, or just re-runs the same processing steps it already does in a single pass — and what the corpus says about when recurrence buys real reasoning power.
This explores whether recurrent transformers — models that loop their layers repeatedly instead of stacking more of them — invent genuinely new operations, or just replay the inference steps a normal feedforward pass already performs. The sharpest answer in the corpus is sobering: when researchers looked inside looped models mechanistically, they found each recurrent cycle converging to a stable fixed point that mirrors and repeats the same stages a feedforward model runs, rather than computing anything new How do looped transformer layers actually behave during inference?. By that reading, the loop is a depth-amplifier, not a new kind of thinking.
But "genuinely new" depends on what you measure. Looped models with shared parameters across iterations achieve compositional generalization and depth extrapolation that vanilla transformers simply cannot — generalizing to unseen combinations of knowledge through a sharp three-phase grokking process (memorize, then in-distribution, then out-of-distribution) Can looped transformers generalize to unseen knowledge combinations?. So even if recurrence re-enacts familiar stages step-by-step, chaining those stages dynamically lets the model reach answers a fixed-depth network is provably barred from. The Hierarchical Reasoning Model pushes this further: by coupling slow abstract planning with fast detailed computation across two timescales, a 27M-parameter recurrent model solves Sudoku and mazes where chain-of-thought collapses — explicitly escaping the AC0/TC0 complexity ceiling that caps fixed-depth transformers Can recurrent hierarchies achieve reasoning that transformers cannot?. That looks less like replay and more like reaching computations the base architecture can't.
The deeper question hiding here is whether transformers ever learn systematic *rules* or only memorize the shape of computations they've seen. There's strong evidence for the pessimistic view: transformers tend to reduce compositional reasoning to linearized subgraph matching — stitching together computation paths memorized from training, then failing badly on novel compositions Do transformers actually learn systematic compositional reasoning?. If that's the baseline, then recurrence faces a hard ceiling — looping a pattern-matcher more times doesn't make it a rule-follower. Yet there's a Turing-completeness result showing a single finite transformer *can* in principle compute any computable function given the right prompt — the capacity is there; standard training just rarely finds it Can a single transformer become universally programmable through prompts?.
That gap between latent capacity and what training actually produces is where the most interesting work lives — and it suggests the answer isn't fixed by architecture alone. Self-improving transformers achieve *exponential* length generalization (10-digit to 100-digit addition) purely by generating solutions, filtering for correct ones, and retraining Can transformers improve exponentially by learning from their own correct solutions?. RL-finetuned models develop genuine in-context reinforcement learning, solving unseen problems within an episode with no weight updates at all Can transformers learn to solve new problems within episodes?. And feedback loops that let a model attend to its own latents grow emergent working memory for unbounded inputs without adding a single weight Can models learn working memory by attending to their own latents?. The pattern across these: the new computation often comes not from the recurrence itself but from what the loop lets the model *accumulate* — state, memory, self-generated training signal.
So the honest synthesis is a split decision. Mechanistically, a loop re-enacts known inference stages How do looped transformer layers actually behave during inference?. Behaviorally, recurrence breaks complexity barriers and unlocks compositional generalization that fixed depth cannot reach Can recurrent hierarchies achieve reasoning that transformers cannot? Can looped transformers generalize to unseen knowledge combinations?. The resolution is that "new computation" lives less in any single re-enacted step and more in the dynamic depth, accumulated state, and self-improvement loops that recurrence makes possible — which is why the same architecture looks like a copy machine under a microscope and a reasoning breakthrough on a benchmark.
Sources 8 notes
Mechanistic analysis reveals looped models converge each recurrent cycle to distinct fixed points, with attention behavior stabilizing across iterations. Recurrent blocks learn to mirror and repeat the same inference stages as feedforward models rather than compute genuinely new operations.
Recurrent-depth transformers with shared parameters across iterations enable systematic generalization and depth extrapolation that vanilla transformers cannot achieve. This emerges through a sharp three-phase process: memorization, in-distribution, then out-of-distribution generalization.
The Hierarchical Reasoning Model couples slow abstract planning with fast detailed computation across two timescales, achieving near-perfect performance on Sudoku and mazes where chain-of-thought methods fail completely. With only 27M parameters and 1,000 samples, HRM escapes the AC0/TC0 complexity ceiling that constrains fixed-depth transformers.
Research shows transformers succeed on in-distribution tasks by memorizing computation subgraphs from training data, not by learning systematic rules. They fail drastically on novel compositions, with errors compounding across reasoning steps.
Research proves a single finite-size transformer exists that can compute any computable function given the right prompt, achieving complexity bounds nearly matching unbounded models. However, standard training rarely produces models that learn to implement arbitrary programs this way.
Standard transformers generalize from 10-digit to 100-digit addition by repeatedly generating solutions, filtering for correctness, and retraining—showing exponential (not linear) out-of-distribution improvement across rounds without saturation.
Llama 3.1 8B fine-tuned with RL exhibits emergent in-context reinforcement learning, solving unseen problems through within-episode adaptation at human-level sample efficiency. This meta-learning emerges from RL's training pressure combined with the transformer's context window, without weight updates.
TransformerFAM demonstrates that adding a feedback loop lets transformers attend to their own latent representations, fostering emergent working memory for indefinitely long inputs. The approach requires no additional weights and improves long-context performance at 1B, 8B, and 24B scales.