Which finetuning method works best across different task and data regimes?
This explores whether there's a single best finetuning method — and the corpus's answer is that 'best' depends on your model size, your task, and what you're optimizing for, with a recurring tension between methods that update weights and methods that leave them alone.
This explores whether one finetuning method wins across the board — and the corpus's clearest finding is that the question itself is mis-framed: the right method depends on your base model size, the task's reward structure, and whether you care more about preserving knowledge or changing behavior. The most fundamental result is that finetuning follows a multiplicative scaling law where a larger base model helps far more than more finetuning data or more tunable parameters How should finetuning scale with model and data size?. So before picking a method, the lever that matters most is what you're starting from.
A striking thread is that the lightest-touch methods often win. Representation finetuning (ReFT) edits frozen hidden states instead of weights and beats LoRA by 10–50x on parameter efficiency across reasoning, instruction-following, and NLU Can editing hidden representations beat weight updates for finetuning?. Proxy-tuning goes further and never touches weights at all, steering outputs at decoding time — and it actually surpasses direct finetuning on knowledge tasks because direct weight updates corrupt knowledge stored in lower layers Can decoding-time tuning preserve knowledge better than weight fine-tuning?. That damage isn't a bug you can tune away: pretraining and finetuning live in different parts of the network, with pretraining scale driving factual accuracy and finetuning scale driving helpfulness Do pretraining and fine-tuning scale independently in language models?. If you finetune hard for behavior, you risk eroding the facts.
The method also has to match the task regime. Preference tuning (RLHF) reduces output diversity in code, where convergence on correct answers is rewarded, but increases it in creative writing, where distinctiveness pays — the same method flips direction depending on the domain Does preference tuning always reduce diversity the same way?. Reinforcement learning carries its own caveats: it tends to collapse onto a single dominant output format inherited from pretraining within the first epoch Does RL training collapse format diversity in pretrained models?, and on reasoning tasks it often sharpens memorized template-matching rather than installing genuine procedures, as out-of-distribution tests expose Do fine-tuned language models actually learn optimization procedures?. So RL is powerful for shaping behavior but a poor bet if you want transferable reasoning.
For multi-task settings the failure mode is interference, and the corpus's answer is structural rather than algorithmic: isolating each task's core parameters and freezing them while merging the rest beats standard multi-task finetuning Can isolating task-specific parameters prevent multi-task fine-tuning interference?. And what you feed the method matters as much as the method — instruction tuning largely teaches the output format, not task understanding, since even semantically empty instructions get comparable results Does instruction tuning teach task understanding or output format?, while teacher-refined data backfires when it overshoots the student model's learning frontier, even if it's objectively higher quality Does teacher-refined data always improve student model performance?.
The takeaway you didn't know you wanted: there's no universal champion, but there is a default worth reaching for — start from the largest base model you can, and prefer interventions that leave pretrained weights intact (representation editing, decoding-time proxies) unless the task specifically rewards reshaping behavior. Match the heavier methods to the regime, not the other way around.
Sources 10 notes
Systematic experiments across 1B–16B models reveal finetuning follows a power-based multiplicative scaling law. Larger base models improve finetuning more than more pretraining data, while increasing PET parameters provides minimal benefit.
ReFT learns task-specific interventions on frozen model representations rather than updating weights, with LoReFT (low-rank linear subspace variant) dramatically outperforming LoRA across reasoning, instruction-following, and NLU benchmarks while using far fewer parameters.
Proxy-tuning closes 88-91% of the alignment gap while surpassing direct fine-tuning on knowledge tasks by leaving base model weights untouched. Direct fine-tuning corrupts knowledge storage in lower layers, whereas proxy-tuning applies distributional shifts that primarily affect reasoning and style.
Emulated Fine-Tuning reveals that scaling pretraining improves factual knowledge while scaling fine-tuning improves behavioral helpfulness. This decoupling has architectural roots: pretraining enriches lower-layer knowledge storage, while fine-tuning modifies upper-layer behavior expression.
RLHF reduces lexical-syntactic diversity in code generation but increases it in creative writing. The direction depends on what each domain incentivizes: code rewards convergence toward correct solutions, while creative writing rewards stylistic distinctiveness.
Controlled experiments show RL consistently amplifies one format distribution from pretraining within the first epoch while collapsing alternatives. The winning format depends on model scale, not necessarily performance, and is largely hidden when starting from proprietary pretrained models.
Even GRPO-trained models show sharp performance drops on out-of-distribution variants (N-1 test sets) compared to in-distribution problems, indicating RL optimizes template-matching rather than genuine problem-solving procedures.
Research shows that identifying core parameter regions per task, clustering overlapping tasks, and freezing core parameters while geometrically merging non-core parameters consistently outperforms standard multi-task fine-tuning. Temporal task scheduling alone proves insufficient without explicit structural parameter isolation.
Models trained on semantically empty or deliberately incorrect instructions achieve comparable performance to those trained on full correct instructions, achieving 43% vs random baseline 42.6%. The semantic content of instructions appears largely irrelevant; what transfers is knowledge of the output space.
Teacher-refined data degrades performance when it exceeds the student's learning frontier, even if objectively higher quality. Students should filter refinements using their own statistical profile to retain only compatible improvements.