Can diverse mediocre traces outperform redundant expert traces?

Standard RLVR rewards a trace for reaching the right answer. SPIRAL introduces set reinforcement learning, which shifts the unit of reward from the individual trace to the sampled set: a group of parallel traces is rewarded for being collectively useful to an aggregator that will condition on all of them. The objective is not "each trace should be correct" but "this set should give the aggregator the raw material to produce a better final answer than any single trace."

This is a meaningful decoupling because a set of individually-mediocre-but-diverse traces can beat a set of individually-strong-but-redundant ones. If every sampled trace commits to the same reasoning path, aggregation is vacuous — the aggregator has nothing to arbitrate. This connects directly to Does policy entropy collapse limit reasoning performance in RL?: standard RL drives traces toward a single dominant format, which is exactly the redundancy that makes aggregation pointless. Set RL supplies a group-level pressure toward complementarity that per-trace reward cannot, therefore preserving the diversity aggregation needs.

The design echoes a broader pattern — that quality at the group level is a different objective than quality at the item level. Where a diversity-aware RL like DARLING adds a semantic-diversity term, SPIRAL bakes complementarity into the reward's unit rather than a bonus term, because the downstream consumer is a trained aggregator whose success defines what "collectively useful" means.