The apparent mismatch between transformers' architectural elegance and their struggles with sequential state tracking has become a focal point in foundation model design—researchers have increasingly documented how purely feedforward stacks force evolving representations deeper with each step, creating a kind of topological gridlock. This tension invites a natural question: if recurrent and looped architectures can address this depth exhaustion, what specifically do they compute differently? Recent work has shown that looped transformer layers re-apply inference stages rather than discover genuinely new computation, while other evidence suggests that tiny recursive networks recursing on latent reasoning states can outperform billion-parameter feedforward models on hard reasoning. Yet the field remains uncertain whether recurrence's advantage lies in implicit activation dynamics, in iterative latent computation during pretraining, or in some hybrid that blends memory efficiency with expressive depth. The question now is whether modern foundation models would benefit more from retrofitting recurrence into existing architectures, or from rethinking the fundamental unit of computation altogether.
Abstract
Transformers encode structure in sequences via an expanding contextual history. However, their purely feedforward architecture fundamentally limits dynamic state tracking. State tracking -- the iterative updating of latent variables reflecting an evolving environment -- involves inherently sequential dependencies that feedforward networks struggle to maintain. Consequently, feedforward models push evolving state representations deeper into their layer stack with each new input step, rendering information inaccessible in shallow layers and ultimately exhausting the model's depth. While this depth limit can be bypassed by dynamic depth models and by explicit or latent thinking that externalizes state representations, these solutions are computationally and memory inefficient. In this article, we argue that temporally extended cognition requires refocusing from explicit thought traces to implicit activation dynamics via recurrent architectures. We introduce a taxonomy of recurrent and continuous-thought transformer architectures, categorizing them by their recurrence axis (depth versus step) and their ratio of input tokens to recurrence steps. Finally, we outline promising research directions, including enhanced state-space models and coarse-grained recurrence, to better integrate state tracking into modern foundation models.