Tacking a 'be diverse' bonus onto an AI's training reward sounds equivalent to designing diversity into the goal — but it probably isn't.

This explores two routes to diversity in RL training: adding an explicit diversity bonus to the reward signal, versus restructuring the objective so diversity falls out of what's being optimized. The corpus suggests these are not equivalent, and the difference comes down to where the diversity is *grounded*. A bonus is an external regularizer — you're paying the model to be different, which raises the question of different in ways that matter or different for its own sake. A set-level or vector-valued objective doesn't pay for diversity at all; it just refuses to reward redundancy, so spread emerges because the task itself rewards complementarity.

The cleanest 'natural' cases are set-level and vector reinforcement. Set RL rewards a *batch* of traces for being collectively useful rather than individually correct, and finds that diverse mediocre traces beat redundant strong ones because an aggregator needs raw material to arbitrate, not confirmation Can diverse mediocre traces outperform redundant expert traces?. Vector-valued rewards get there a different way: by decomposing reward per test-case, criterion, or persona and keeping it unscalarized, solutions specialize along a Pareto frontier, producing 'competent diversity grounded in real task trade-offs rather than external regularizers' Can reward vectors be the hidden source of solution diversity?. In both, diversity is a structural property of the objective, not a term you tune.

The bonus route is not a failure, though — it just carries the burden the set-level objective sidesteps. DARLING optimizes quality *and* semantic diversity jointly using a learned classifier, and finds that a diversity reward can actually catalyze exploration and yield higher-quality outputs than a quality-only baseline Can diversity optimization improve quality during language model training?. So a well-designed bonus can do more than preserve spread — it can improve quality. The catch is that a bonus is only as good as the thing measuring 'diverse.' A learned semantic classifier grounds it; a naive lexical bonus would reward superficial variation. That's why the framing matters: the set-level objective never has to answer 'diverse according to what,' because collective usefulness answers it implicitly.

The reason any of this is needed shows up in what RL does by default: it *squeezes* exploration diversity, with search agents collapsing onto narrow reward-maximizing strategies through the same entropy-collapse mechanism seen in reasoning Does reinforcement learning squeeze exploration diversity in search agents?. Both bonuses and set-level objectives are counter-pressures against that collapse — and so are approaches that don't touch the reward at all, like step-level critique in the training loop, which counteracts tail-narrowing and preserves solution diversity across self-training iterations Do critique models improve diversity during training itself?. That's a third answer to the question: you can get diversity without either a bonus or a set objective, by intervening in the training dynamics directly.

So can a diversity bonus achieve what set-level objectives achieve naturally? Roughly, yes on outcomes, but with an extra design liability. There's also a subtler warning from the multi-objective literature: when you scalarize competing objectives with fixed weights, you invite reward hacking and mis-weighting — which is why variance-based weighting up-weights high-signal objectives automatically instead of using tuned constants How should multiple reward objectives be weighted during training?, and why rubrics work better as accept/reject *gates* on rollout groups than as dense reward terms Can rubrics and dense rewards work together without hacking?. The through-line: the more diversity lives inside the structure of the objective rather than in a bolted-on scalar bonus, the less it can be gamed — and that structural grounding, not the diversity itself, is what set-level objectives give you for free.