Can algorithm choice like PPO substitute for recipe-level design decisions?
This explores whether picking a particular RL algorithm (like PPO vs. GRPO vs. DAPO) is the thing that determines results — or whether the real leverage lives in the surrounding 'recipe': the data, the reward shape, the loss aggregation, and the pretrained model you start from.
This explores whether the choice of RL algorithm is what actually drives performance, or whether it's a relatively interchangeable knob compared to the design decisions around it. The corpus answers fairly bluntly: algorithm choice is mostly *not* where the leverage is. One study shows that plain critic-free PPO can match or surpass more elaborate methods like GRPO and DAPO once you add two recipe-level ingredients — advantage normalization and token-level loss aggregation — and concludes that the pretrained prior, not the optimizer, sets the performance ceiling Can two simple techniques match complex RL algorithms?. A companion finding makes the deeper claim explicit: Expert Iteration, PPO, and RC-RL all perform comparably on reasoning because exploration is bounded by the pretrained distribution. RL here acts as *selection*, not *discovery* — the solutions are already latent in the prior, and the algorithm just surfaces them Does the choice of RL algorithm actually matter for reasoning?.
If the optimizer is interchangeable, what isn't? The corpus points repeatedly to reward and loss design as the real recipe. Binary correctness rewards quietly degrade calibration by rewarding confident guessing — and the fix is a recipe change (adding a Brier-score term), not an algorithm swap Does binary reward training hurt model calibration?. How you *use* a rubric matters more than which RL method consumes it: rubrics as accept/reject gates prevent reward hacking, while the same rubrics converted into dense scores invite it Can rubrics and dense rewards work together without hacking?. Even the shape of the loss function trades off against itself — utility-weighted loss sharpens decisions but starves representation learning, so the design decision (symmetric loss, adjust post-hoc) beats the seemingly clever objective Can utility-weighted training loss actually harm model performance?.
There's a nice unifying reframe here: alignment methods may work for reasons that have nothing to do with the algorithm's surface mechanics at all. KTO argues that DPO and PPO-Clip succeed because they implicitly mirror prospect theory — human loss aversion — which is why a simple binary utility signal can outperform pairwise preferences when the pretrained model is strong Why do alignment methods work if they model human irrationality?. The 'algorithm' is almost incidental to the underlying decision-theoretic structure it's modeling.
The corpus also nudges you to question whether some of the things algorithms supposedly trade off are even real. The exploration-exploitation tension that motivates a lot of algorithm design turns out, under hidden-state analysis, to be largely a token-level measurement artifact rather than a fundamental constraint Is the exploration-exploitation trade-off actually fundamental?. Meanwhile, gains that look algorithmic often come from structure instead: Tree-GRPO manufactures process-level supervision purely from branching rollout structure, no new optimizer or reward model required Can tree structure alone convert outcome rewards into process supervision?. And effects you might attribute to a method are really domain-dependent — RLHF reduces diversity in code but increases it in creative writing, because the recipe's *target* domain decides the outcome Does preference tuning always reduce diversity the same way?.
So the honest answer is: no, algorithm choice mostly can't substitute for recipe-level design — but the more interesting takeaway is the inversion. The corpus suggests the causal weight runs the other way. If the pretrained prior caps what any optimizer can find, then your highest-leverage moves are upstream and around the algorithm — the base model, the reward structure, the loss shape, the data domain — and the choice between PPO and its fancier cousins is closer to a formatting decision than a strategic one.
Sources 9 notes
Advantage normalization and token-level loss aggregation allow critic-free PPO to surpass more complex algorithms. Systematic evaluation shows most RL techniques are setup-sensitive; the pretrained prior, not algorithm choice, sets performance ceiling.
Expert Iteration, PPO, and RC-RL perform comparably on reasoning because exploration is constrained by the pretrained distribution, not the optimizer. RL functions as selection, not discovery—the prior contains most solutions the algorithm will find.
Binary correctness rewards incentivize high-confidence guessing because they don't penalize confident wrong answers. Adding the Brier score as a second reward term mathematically guarantees joint optimization of accuracy and calibration without trade-off.
DRO shows that using rubrics to accept or reject rollout groups—rather than converting rubric scores into dense rewards—prevents reward hacking. This separation preserves the categorical strength of rubrics while letting token-level rewards optimize within valid answers.
Asymmetric loss functions correctly incentivize choosing but degrade representation learning by reducing gradient signals for substantive feature acquisition. Training with symmetric loss then adjusting predictions post-hoc outperforms direct utility-weighted training on the same utility objective.
KTO formalizes what DPO and PPO-Clip do implicitly: they succeed because they mirror prospect theory's structure of human decision-making. Binary utility signals suffice and outperform pairwise preferences when pretrained models are strong.
Hidden-state analysis using Effective Rank metrics shows near-zero correlation between exploration and exploitation, revealing the trade-off emerges only at token level. VERL demonstrates simultaneous enhancement achieving 21.4% accuracy gains on Gaokao 2024.
Tree-GRPO uses branching structure to transform trajectory-level outcome rewards into step-level preference signals through sibling subtree comparison, eliminating the need for separate process reward models or step-level annotation while scaling with computational budget.
RLHF reduces lexical-syntactic diversity in code generation but increases it in creative writing. The direction depends on what each domain incentivizes: code rewards convergence toward correct solutions, while creative writing rewards stylistic distinctiveness.