If an AI training trick boosts scores by 23 points on one task set, does that gain carry over when the tasks change?
Does the 23-point improvement replicate across different sets of training tasks?
This explores whether a specific reported gain (a 23-point improvement) still holds when a model is trained on a different set of tasks. The corpus doesn't say which result the 23 points comes from, so this answer covers the broader question of when training gains survive a change of tasks.
This explores whether a specific reported gain (a 23-point improvement) still holds when a model is trained on a different set of tasks. One caveat first: none of the notes retrieved here mention a 23-point figure, so the corpus can't confirm or rule out that particular result. What it does have is a lot of evidence on a closely related question: when a training gain shows up on one set of tasks, how often does it survive a change of tasks? The short answer is less often than the headline numbers suggest, and the reasons are specific and worth knowing.
The most direct warning is that the training data on its own isn't what you'd be replicating. One note argues that the reusable unit of reasoning training is a whole feedback setup, not a dataset. That setup includes the verifier, the base model, its lineage, the optimizer, the scaffold and the compute budget. Change any one of them and the same data can produce a different effect What is the actual reusable unit of reasoning data?. Training order matters too. Putting structured tasks such as math before open-ended creative tasks gave a 6.2% gain over training on everything at once, because structured training narrows the model's outputs and can damage creative ability if it comes first Does training order reshape how models handle different task types?. So a gain measured on one task mix may partly come from how those particular tasks interact with each other.
There are also reasons a big number might not reflect real learning at all. A math-specialized model could reconstruct 54.6% of a popular benchmark from partial prompts but scored 0% on a newer test released after its training. Gains from reinforcement learning with verifiable rewards (RLVR) on contaminated benchmarks turned out to be mostly memorization Does RLVR success on math benchmarks reflect genuine reasoning improvement?. Other notes find that training often teaches surface form rather than skill. Instruction tuning on deliberately wrong instructions did about as well as tuning on correct ones (43% vs. 42.6%) Does instruction tuning teach task understanding or output format?. Imitation training copies a stronger model's confident style without closing the capability gap Can imitating ChatGPT fool evaluators into thinking models improved?. RL tends to lock onto one output format inherited from pretraining within the first epoch Does RL training collapse format diversity in pretrained models?. Gains of these kinds tend to vanish when the tasks change.
The corpus also shows what a convincing replication check looks like. AIDE2 tested its improvements on four held-out benchmarks, including physics-based weather forecasting, which lay outside the tasks used to develop it, and the gains carried over Do AIDE2's improvements transfer to unseen tasks?. That is the standard to look for. Whatever the 23-point result is, it holds up only if the paper reports it on tasks the method never saw during selection, and ideally with a different base model.
The less obvious lesson is that the question may be framed too narrowly. Changing the training tasks is only one of several things that can make a gain disappear. Changing the base model, the verifier or the order of training can do the same. If you can find the original note behind the 23-point claim, check which of those factors the authors held fixed.
Sources 7 notes
The reusable unit in post-training is a feedback interface entangled with six factors: verifier, base model, lineage, optimizer, scaffold, and budget. Changing any one alters the same data's effect, making attribution tractable only when these are jointly released.
Omni-Thinker shows structured domains decrease output entropy while creative domains increase it. BWT-guided scheduling—training structured tasks first—yields 6.2% gains over joint training by preventing entropy collapse from damaging open-ended capabilities.
Qwen2.5-Math-7B reconstructs 54.6% of MATH-500 from partial prompts but scores 0.0% on post-release LiveMathBench, revealing dataset contamination. On clean benchmarks, only correct rewards improve performance; random and inverse rewards fail or degrade reasoning ability.
Models trained on semantically empty or deliberately incorrect instructions achieve comparable performance to those trained on full correct instructions, achieving 43% vs random baseline 42.6%. The semantic content of instructions appears largely irrelevant; what transfers is knowledge of the output space.
Imitation models fool human evaluators by mimicking ChatGPT's confident, fluent style while failing to improve factuality or generalization on novel tasks. The ceiling is set by base model capability, not fine-tuning method—better fundamentals, not shortcuts, drive real improvement.
Show all 7 sources
Controlled experiments show RL consistently amplifies one format distribution from pretraining within the first epoch while collapsing alternatives. The winning format depends on model scale, not necessarily performance, and is largely hidden when starting from proprietary pretrained models.
The paper reports that AIDE2's improvements transfer to four held-out benchmarks spanning machine learning, algorithm engineering, and physics-based weather forecasting—the last being outside the selection task distribution. This demonstrates transferable gains beyond overfitting to the selection set.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Planted in Pretraining, Swayed by Finetuning: A Case Study on the Origins of Cognitive Biases in LLMs
- Sharpening Tax in Post-Training
- Reinforcement Learning for Reasoning in Large Language Models with One Training Example
- Local Coherence or Global Validity? Investigating RLVR Traces in Math Domains
- On the Interplay of Pre-Training, Mid-Training, and RL on Reasoning Language Models
- Eliciting Reasoning in Language Models with Cognitive Tools
- A Primer in Post-Training Reasoning Data: What We Know About How It Works
- The False Promise of Imitating Proprietary LLMs