Can preference learning fix the rigid output format problem better than supervised training?
This explores whether preference-based training (RL, reward models) handles the 'output format' problem differently than supervised fine-tuning — and the corpus suggests the framing itself hides a trap: both methods may be doing format-fitting rather than teaching the task.
This reads the question as: if supervised fine-tuning mostly teaches a model *what shape* the answer should take, can preference learning do something deeper or better? The corpus complicates the premise in a productive way. The most direct finding is that supervised instruction tuning isn't really teaching task understanding at all — models trained on semantically empty or even deliberately wrong instructions land within a fraction of a point of models trained on correct ones (43% vs. a 42.6% baseline). What transfers is knowledge of the output space, not comprehension Does instruction tuning teach task understanding or output format?. So 'rigid output format' isn't a side effect of SFT — it may be most of what SFT actually does.
The twist is that preference learning doesn't obviously escape this. When RL is applied on top of a pretrained model, it tends to *collapse* format diversity rather than expand it — converging on a single dominant pretraining format within the first epoch and suppressing the alternatives, with the 'winner' determined by model scale rather than by which format performs better Does RL training collapse format diversity in pretrained models?. If your complaint is rigidity, naive RL can make it worse, not better. And reward models carry their own version of the format trap: efforts to train away undesirable 'persona distortions' in AI writing succeeded at reducing the distortions but also reduced what writers liked, because the desirable qualities ran through the same generative tendencies as the unwanted ones Can AI writing assistance remove distortion without losing appeal?. Preference signals don't cleanly separate 'good format' from 'good substance.'
Where preference learning does pull ahead is when the reward is decomposed rather than holistic. Breaking instruction-following into verifiable sub-criteria — checklists — improves performance precisely because it 'reduces overfitting to superficial artifacts that plague holistic reward models' Can breaking down instructions into checklists improve AI reward signals?. In other words, the advantage isn't 'preference vs. supervised' in the abstract; it's whether the training signal targets the right thing or just rewards surface form. Rewarding real downstream outcomes — recommendation metrics like NDCG and Recall used directly as RL rewards — sidesteps SFT distillation entirely and ties the model to task success rather than format mimicry Can recommendation metrics train language models directly?.
There's a deeper reason preference signals are slippery, and it's upstream of the method: the annotations that feed reward models aren't one clean signal. They decompose into genuine preferences, non-attitudes, and constructed-on-the-spot preferences, distinguishable only by consistency across measurement conditions — and treating them uniformly contaminates the reward model Do all annotation responses measure the same underlying thing?. So 'preference learning' inherits whatever noise lives in human labels, which can itself be format-shaped rather than substance-shaped.
The genuinely interesting answer, then, isn't yes or no — it's that the better contrast is *holistic vs. decomposed/verifiable* signal, not *preference vs. supervised*. The most promising thread here may be moving the evaluator inside the model: post-completion learning trains a model to compute its own reward in the unused sequence space after its output, internalizing self-assessment instead of leaning on an external reward model Can models learn to evaluate their own work during training?. That reframes the whole problem — if rigidity comes from optimizing toward a fixed external target, a model that learns to judge its own substance is a different lever than either SFT or preference learning as usually posed.
Sources 7 notes
Models trained on semantically empty or deliberately incorrect instructions achieve comparable performance to those trained on full correct instructions, achieving 43% vs random baseline 42.6%. The semantic content of instructions appears largely irrelevant; what transfers is knowledge of the output space.
Controlled experiments show RL consistently amplifies one format distribution from pretraining within the first epoch while collapsing alternatives. The winning format depends on model scale, not necessarily performance, and is largely hidden when starting from proprietary pretrained models.
Training reward models successfully reduced measured persona distortions, but also reduced writer acceptance of the output. This suggests desirable properties like clarity and confidence operate through the same generative tendencies that produce problematic distortions.
RLCF and RaR methods decompose instruction quality into verifiable sub-criteria, improving performance on benchmarks like FollowBench and HealthBench. This decomposition principle reduces overfitting to superficial artifacts that plague holistic reward models.
Rec-R1 demonstrates that LLMs can be trained directly on rule-based recommendation metrics like NDCG and Recall as RL reward signals, eliminating the need for SFT distillation from proprietary models while remaining model-agnostic across different retriever architectures.
Behavioral science reveals that annotations contain genuine preferences, non-attitudes, and constructed preferences—distinguishable by consistency across measurement conditions. Treating them uniformly contaminates reward model training and downstream alignment.
Post-Completion Learning exploits unused sequence space after model output to train self-assessment capabilities during training while maintaining zero inference cost. The model learns to compute its own reward functions, internalizing evaluation rather than relying on external reward models.