SPIRAL: Learning to Search and Aggregate

Language model reasoning can be substantially improved at test time via scaffolds that scale inference compute across different primitives—sequential reasoning within a trace, independently sampled parallel traces, and aggregation of multiple reasoning traces into a final response. During post-training, however, language models are optimized only for sequential reasoning within a single trace. We introduce Sequential-Parallel-Aggregative Reinforcement Learning (Spiral), a framework in which a language model is trained to use all three primitives, as part of a unified inference compute pipeline. Concretely, the language model first samples a set of independent traces in parallel, each produced through sequential chain-of-thought reasoning, and then generates a final aggregation trace conditioned on those traces; all components are optimized end-to-end against the reward of the final aggregated response. To train this system, Spiral uses set reinforcement learning to teach models to produce a set of traces that are collectively useful for an aggregator and standard reinforcement learning to teach models to aggregate the set into improved final responses.

Introduction. Language models exhibit a distinct jagged edge in intelligence during open-ended discovery. When inference compute is scaled, these systems can resolve highly complex challenges, such as the 70-year-old open Erdős unit distance problem [ABG+26]. Yet, on other rigorous tasks, such as certain First Proof problems [ABH+26, ASWW26], models struggle to make progress as their performance fails to scale with additional compute. Crucially, these failures do not occur because the unsolved problems are strictly more difficult, but because of an inability to effectively utilize the inference compute available. This stagnation highlights a critical bottleneck. When sequential compute is scaled—allowing the model to allocate thinking tokens before producing a final answer—models frequently misallocate this budget, extensively detailing routine operations while glossing over complex, decisive logical leaps [ASWW26].

Discussion / Conclusion. We introduced Sequential-Parallel-Aggregative Reinforcement Learning (Spiral), a framework that combines set reinforcement learning with standard reinforcement learning to train a model to effectively use sequential, parallel, and aggregative inference compute. Our early results suggest that jointly training a model to use these primitives enables it to better search, verify, and refine its generations, leading to improved scaling of performance at test time.