Does dataset size matter more than dataset fit?

Because QLoRA made finetuning cheap enough to train "more than 1,000 models" across datasets, architectures, and scales from 80M to 65B parameters, Dettmers and colleagues could compare instruction-tuning datasets directly rather than by reputation. They report: "data quality is far more important than dataset size, e.g., a 9k sample dataset (OASST1) outperformed a 450k sample dataset (FLAN v2, subsampled) on chatbot performance, even when both are meant to support instruction following generalization." A second, related finding sharpens this into a claim about fit rather than quality in the abstract: "strong Massive Multitask Language Understanding (MMLU) benchmark performance does not imply strong Vicuna chatbot benchmark performance and vice versa — in other words, dataset suitability matters more than size for a given task."

The paper's own explanation is distributional similarity between the finetuning data and the benchmark, not an intrinsic quality ranking of datasets. In the Limitations section it states that "the performance of these benchmarks likely depends how similar the finetuning data is to the benchmark dataset. For example, FLAN v2 is similar to MMLU, but dissimilar to chatbot benchmarks and vice versa for the Chip2 dataset and both models score accordingly on the MMLU and Vicuna benchmarks." Rankings were produced through tournament-style comparison judged by GPT-4 and human annotators and aggregated into Elo scores, with the paper noting the two judges "largely agree" on model ranking but sometimes disagree strongly — a caveat on how the chatbot-performance numbers behind this finding were themselves measured.

This converges with, but is mechanistically distinct from, Can we prune training data without hurting model performance?. That note's claim operates within a single dataset: ranking individual examples by difficulty (EL2N, forgetting, memorization scores) and discarding the redundant easy ones improves pretraining efficiency. QLoRA's finding operates across datasets: OASST1 and FLAN v2 differ in source, topic distribution, and format, not just example count, so the comparison is between whole corpora selected for benchmark fit rather than between curated and uncurated subsets of the same corpus. Both findings land on the same broader principle — that scaling dataset size is not the lever that matters most — but one is about which examples to keep and the other is about which corpus to choose.

What the excerpt does not establish is causation isolated from size: OASST1 and FLAN v2 differ in content and origin as well as in sample count, so the "9k beats 450k" comparison does not control for size as the sole variable, and the paper frames this as an illustrative example rather than a controlled ablation. The authors also caution more broadly that "current chatbot benchmarks are not trustworthy to accurately evaluate the performance levels of chatbots," which bears directly on how much weight the Vicuna-benchmark ranking behind this finding can carry. The implication the paper draws — that choosing instruction data suited to the target task beats simply acquiring more data — is reasonable at the strength of a documented example within one evaluation setup, not a benchmark-independent law about dataset size.