Why do intermediate predictors in looped models align with final outputs?
This explores why, in models that loop the same layers over and over, the predictions made at intermediate steps tend to agree with the final answer — and what that says about what looping actually computes.
This explores why, in models that loop the same layers over and over, the predictions made at intermediate steps tend to agree with the final answer. The short version the corpus suggests: a looped model isn't inventing brand-new computation on each pass — it's settling into a stable trajectory, and the intermediate states are already pointed at where the final state lands.
The clearest account comes from work showing that looped transformers converge to learned cyclic fixed points How do looped language models actually improve reasoning in depth?. Instead of doing something different on each loop, each recurrent pass re-enacts the stages a deep feedforward network would have done in separate layers, and the system relaxes toward a stable point it keeps returning to. Because the loop is pulling the hidden state toward that attractor, an intermediate readout is essentially an early snapshot of the same destination — which is exactly why intermediate predictors line up with the final output rather than wandering off and being corrected later.
That framing also explains a surprising empirical wrinkle: more looping is not monotonically better. One study finds a sweet spot at two loops, where the second loop carries the real refinement, while three or more loops regress and start to oscillate with less representational diversity Does adding more loops always improve looped language models?. If the model has already converged near its fixed point, extra iterations don't add new information — they perturb an answer that was effectively settled, which is the mirror image of why early intermediate states already agree with the end.
The deeper reason looping works at all is that recursion buys you state-tracking and compositional generalization that simply adding parameters does not Can models learn by looping instead of growing larger?. Re-applying a shared block lets the model carry and refine a running state, and convergence itself acts as a natural halting signal — the alignment between intermediate and final predictions is the model telling you it has stopped changing its mind. The same logic scales beyond language: in world models, iterative latent depth refines an environment state step by step, spending more passes on harder predictions, with stability guarantees that keep the refinement from diverging Can looped computation replace parameter count in world models?.
What you didn't know you wanted to know: this means the intermediate–final agreement is a feature you can read as a confidence/halting cue, not just an artifact. When a looped model's early guess already matches its last one, the loop has converged; when intermediate predictions keep shifting, it's still climbing toward its fixed point — and pushing past the sweet spot risks shaking loose an answer it had already found.
Sources 4 notes
Each recurrent layer converges to distinct fixed points forming stable cyclic trajectories. Looped models learn to mirror and repeat feedforward inference stages rather than discover new computation, emerging naturally without explicit training.
LoopCoder-v2 shows that two loops deliver broad gains over baseline, but three or more loops regress. Loop 2 carries the productive refinement; later loops oscillate with reduced representational diversity rather than converging toward better performance.
Models that re-apply layers in recurrent depth outperform larger feedforward networks on reasoning tasks. This works because recursion enables state tracking and compositional generalization that parameter scaling alone cannot achieve, with convergence signals providing natural halting.
LoopWM achieves up to 100x parameter efficiency by refining latent environment states through iterative computation in a shared block, with spectral-norm constraints providing formal stability guarantees. The approach mirrors physical system recurrence, spending more depth on harder prediction steps.