Loop the Loopies!

We present the Loopie series, consisting of two Mixture-of-Experts (MoE) models: a 20B-parameter model with 2B active parameters and a 6B-parameter model with 0.6B active parameters. Looped Transformers have long faced a challenge: given an N-fold increase in pre-training compute, increasing the parameter count by a factor of N usually outperforms looping a model N times. Loopie addresses this challenge. Extensive ablation studies, including comparisons with a vanilla 30B-A3B model, show that Loopie substantially outperforms vanilla Transformer baselines trained with the same compute budget. With a novel post-training method, Loopie develops strong reasoning abilities and achieves frontier-level reasoning performance.

Introduction. Looped Transformers, first introduced as Universal Transformers (Dehghani et al., 2019), have recently re-emerged as a compelling alternative to conventional depth scaling. Rather than stacking distinct layers, they repeatedly apply the same model across recurrent steps, closely connecting this approach to parameter sharing in Transformers (Lan et al., 2020; Dehghani et al., 2019). This recurrent approach has shown strong empirical performance across a wide range of domains, including language modeling, algorithmic learning, and abstract reasoning (Gao et al., 2025b; Yang et al., 2024; Jolicoeur-Martineau, 2025; Wang et al., 2025; Dehghani et al., 2019; Saunshi et al., 2025; Bae et al., 2025; Geiping et al., 2025; Zhu et al., 2025; Frey et al., 2026a; Huang et al., 2026b). A growing body of work suggests that recurrent computation is especially well suited to complex problems.

Discussion / Conclusion. We introduced Loopie, a family of looped MoE language models that makes recurrent depth competitive under a matched pre-training compute budget. By combining layer-loop recurrence with a hardware-aware scaling recipe, Loopie consistently outperforms compute-matched vanilla Transformer baselines across model scales. A large-scale post-training pipeline based on Supervised Pre-training and reinforcement learning further equips Loopie with strong mathematical reasoning and coding abilities. These results suggest that recurrent computation, when jointly optimized with architecture and training efficiency, can serve as a practical scaling axis for large language models.