When AI training on human feedback wastes data, is the fix capping how much the model changes each step — or fixing what it trains on?

This explores whether trust-region constraints — the guardrails that stop a policy from lurching too far in a single update — can cure RLHF's sample inefficiency. Worth saying plainly up front: the collection doesn't have a paper that pits trust regions directly against sample cost. What it does have is a cluster of findings about *why* RLHF wastes samples, and most of them point somewhere a trust region can't reach. A trust region controls how far you step; it does nothing about whether the ground you're stepping on carries useful signal.

The corpus keeps locating the waste in the reward signal's shape rather than the step size. When within-prompt reward variance is low, the useful gradient washes out and regularization takes over, dragging the policy toward generic, input-agnostic templates — and the fix that works is selecting high-variance prompts *before* you update, not constraining the update (Why do language models collapse into generic templates?). Push the difficulty the other way and it fails too: overly hard samples let group-relative normalization treat rare lucky successes as high-advantage, reinforcing shortcuts and answer-repetition that then contaminate abilities the model already had (Do overly hard RLVR samples actually harm model capabilities?). Both are sample-efficiency failures, and both are about *which* samples you train on — the trust region is silent on that choice.

There's also a quieter collapse that a trust region arguably makes worse, not better. RL post-training tends to converge on a single dominant format inherited from pretraining within the first epoch, suppressing the alternatives — and the winner is picked by scale, not by performance (Does RL training collapse format diversity in pretrained models?). A tight trust region is a commitment to stay near where you already are; if where you already are is one narrow format basin, constraining movement locks the collapse in rather than preventing it. Preference tuning's effect on diversity isn't even uniform — it compresses code toward convergent solutions while expanding creative writing (Does preference tuning always reduce diversity the same way?) — so a single global constraint on step size can't be the right knob across domains that want opposite things.

Where the corpus does gesture at real efficiency gains, they come from changing the *signal*, not fencing the *step*. Using the model's own answer-span confidence as an intrinsic reward builds synthetic preferences that sharpen reasoning and reverse RLHF's calibration damage — no human labels, no external verifier (Can model confidence work as a reward signal for reasoning?). And at a more fundamental level, sample efficiency is shown to be a property of *what you predict against*: predicting your own latents recovers compositional structure with a number of samples flat in depth, while token-level learning needs exponentially more (Why is predicting latents more sample-efficient than tokens?). That's the deeper lesson — efficiency is bought by richer, better-correlated learning targets, a lever that sits entirely outside the trust region's job description.

So the honest answer the collection points to: trust regions are a stability tool, and stability is not the same problem as sample efficiency. They can keep RLHF from blowing up, but the samples RLHF wastes are wasted on low-variance, mis-calibrated, or collapse-prone reward signals — and there's a formal ceiling underneath all of it, since a model can't reliably validate its own improvements without something external in the loop (What stops large language models from improving themselves?). The thing worth walking away with: the fix for sample inefficiency in the corpus is almost never 'move more carefully' — it's 'pick better problems and reward them with a signal that actually carries information.'