Does your finetuning data need to closely resemble the benchmark to predict how well your model will score on it?
How similar must finetuning data be to a benchmark to predict performance?
This explores whether the closeness between your finetuning data and a target benchmark tells you how well the model will score, and how close that match has to be before it matters more than how much data you have.
This explores whether the closeness between your finetuning data and a benchmark predicts how well a model will score, and how close the two need to be. The short answer from the corpus: closeness matters more than size, and it can beat size by a lot. The corpus does not give a fixed threshold, though. It shows that similarity predicts performance. It does not say how much similarity is enough.
The clearest example comes from the QLoRA experiments. Across more than 1,000 finetuned models, a 9,000-example conversational dataset (OASST1) beat a 450,000-example slice of FLAN v2 on chatbot benchmarks. The deciding factor was how closely the training data's style and task mix matched what the benchmark measures Does dataset size matter more than dataset fit?. LESS goes further and makes similarity something you can measure. It compares the gradients of training examples with those of examples from the target task, then keeps only the closest 5% of the data. That 5% beats training on everything. The surprising part is why: some examples in a mixed dataset actively hurt a specific skill by pulling the model toward the wrong reasoning strategy Can we train better models on less data?. So similarity is more than a bonus. Dissimilar data can do harm.
This changes what "similar" means. It is not about shared topic words. It is about whether training pushes the model in the same direction the benchmark rewards. A related finding from RL training shows how strong format pressure can be. Post-training tends to lock onto one dominant format from pretraining within the first epoch and squeeze out the others Does RL training collapse format diversity in pretrained models?. If your benchmark expects a different format from the one training reinforces, a closely matched dataset may still underperform. Scale is a second factor. Finetuning gains grow multiplicatively with base model size more than with extra data How should finetuning scale with model and data size?. Matching the data to the benchmark is one lever, and the starting model is another.
There is a catch. If you optimize your data to look like the benchmark, you are predicting benchmark performance, which may not be what you care about. Several notes warn that benchmark scores transfer poorly. Short-interaction scores fail to predict how models hold up over long delegated workflows Do short benchmarks predict how models perform over long workflows?. A single score hides trade-offs across separate capability axes Does a single benchmark score actually predict agent readiness?. Auto-graded benchmarks favor neatly specified tasks over messy real ones Do automated benchmarks hide what frontier AI systems can really do?. Tightly matching data to a benchmark can raise the score while the model gets no better at the actual job.
The practical takeaway: a small dataset whose gradients point the same way as the target task is a better predictor than a large, loosely related one, and gradient similarity is the most concrete measure the corpus offers. Nobody here has mapped the curve of how much similarity buys how much accuracy. That is an open question, and the evaluation papers suggest the more useful question may be similarity to the real task rather than to the benchmark.
Sources 7 notes
QLoRA trained over 1,000 models across datasets and scales, finding that a 9k-sample OASST1 dataset outperformed a 450k-sample FLAN v2 subset on chatbot benchmarks. The key factor was distributional similarity between training data and target benchmark, not corpus size.
LESS uses low-rank gradient features to select instruction data most similar to target capabilities, and training on the selected 5% consistently outperforms full dataset training. The improvement occurs because mixed datasets contain examples that actively hinder specific skills by shifting reasoning strategy away from task requirements.
Controlled experiments show RL consistently amplifies one format distribution from pretraining within the first epoch while collapsing alternatives. The winning format depends on model scale, not necessarily performance, and is largely hidden when starting from proprietary pretrained models.
Systematic experiments across 1B–16B models reveal finetuning follows a power-based multiplicative scaling law. Larger base models improve finetuning more than more pretraining data, while increasing PET parameters provides minimal benefit.
DELEGATE-52 evaluated models across 50-round-trip relays and found short-interaction performance does not predict sustained delegation accuracy. Models ranking similarly on single-turn tasks diverged dramatically by relay 25, revealing degradation curves invisible to standard benchmarks.
Show all 7 sources
Capability decomposes into task success, privacy compliance, long-horizon retention, mode-shift behavior, and ecosystem readiness. Models ranked highest on one axis often rank lower on others, making single-score evaluations systematically misleading for real deployment.
Automated benchmarks both overstate and understate capability by privileging precisely-specified, auto-gradable tasks. Open-world evaluations of long-horizon messy tasks through qualitative log analysis—with cost explicitly reported—correct these distortions and catch emerging capabilities earlier.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Open-World Evaluations for Measuring Frontier AI Capabilities
- Measuring AI Ability to Complete Long Software Tasks
- Echo Chamber: RL Post-training Amplifies Behaviors Learned in Pretraining
- Planted in Pretraining, Swayed by Finetuning: A Case Study on the Origins of Cognitive Biases in LLMs
- The Art of Scaling Reinforcement Learning Compute for LLMs
- Sharpening Tax in Post-Training
- Exploring Format Consistency for Instruction Tuning
- Agents' Last Exam