Can a stronger model lift a weaker one at test time without retraining?
Whether inference-time harnesses written by capable models can transfer their strengths to weaker models on reasoning tasks without parameter updates, and what mechanisms enable this transfer.
The paper asks whether strong-to-weak transfer, usually done by distillation that updates the smaller model's parameters, can instead happen at test time. A stronger builder model constructs an inference-time harness for a weaker target model and refines it over several rounds against a validation set that is 5% of the data, across four Theory-of-Mind benchmarks. The abstract reports average target-model performance "nearly doubling" from 0.49 to 0.91. The discussion gives the finer picture over 72 runs: a mean uplift of +0.275 over the GPT-5.4-mini no-scaffold baseline, every run and all 11 builder configurations above baseline, and a best scaffold at 0.912 (+0.423). The 0.91 headline therefore appears to be the best scaffold, not the mean. That best scaffold approaches a human-inspired harness reference (0.939) on the same backbone "despite using no human ToM-specific engineering."
The paper's framing is that a small model's failure "may reflect not only insufficient internal capability, but also excessive cognitive load imposed by the way the task is presented." Its analysis locates the gains "primarily" in three moves: offloading unstable model reasoning into deterministic code, benchmark-specific routing, and strict answer-format enforcement. It says they do not come from encouraging the target model to reason more extensively or sample more broadly. The procedure is reported as reproducible but not deterministic: mean within-cell standard deviation is 0.036, roughly an order of magnitude below the uplift, and the remaining variance sits in deterministic-solver strategies where one implementation error can swing a benchmark.
This sits awkwardly beside Can non-reasoning models catch up with more compute?, which finds that more inference compute does not close a training-regime gap. The two agree that extra reasoning or sampling is not the lever. They differ in what fills the gap: here a harness written by a stronger model carries much of the load outside the weights. One reading is that a harness supplies structure rather than manufacturing capability, but the excerpt does not test that. In the How do knowledge injection methods trade off flexibility and cost?, a harness is closest to prompt optimization, since nothing is trained. Yet deterministic code and routing are not prompt content, so it fits that category only loosely. It also echoes Does RL post-training create reasoning or just deploy it? in placing part of the gap in deployment rather than raw capability, though by relocating work out of the model, not by eliciting it.
The excerpt does not establish how far this generalizes. Results cover four Theory-of-Mind benchmarks, and the analysis credits "benchmark-specific routing" as one source of gain, so the excerpt gives no evidence for tasks outside them. It does not show that the target model reasons better, and the stated mechanism suggests it mostly does not. The "validation-efficient" claim is cut off before any detail. At the strength supported, a harnessed weak model's benchmark score reflects the harness as much as the model, and part of the measured weak-to-strong gap on these tasks is a presentation and execution gap that can be closed without touching parameters.
Inquiring lines that read this note 70
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
Do harness improvements deliver capability gains comparable to model scaling?- Which foundation model tiers most benefit from harness updates?
- How do prompt optimization and code harnesses compare for capability transfer?
- What makes a harness low-friction for model strategy?
- Can weaker models match stronger ones by reorganizing harness-side components?
- Do models co-adapt their harnesses to specific executor strengths?
- Can mid-tier models benefit more from harness improvements than frontier models?
- Does harness scaling represent a fundamentally different path than model scaling?
- How much does harness design contribute to reported model capability scores?
- What distinguishes the fast scaffold learning loop from parametric model weight updates?
- How does harness structure affect planner token efficiency compared to model size?
- Can weaker models benefit equally from harness updates as stronger ones?
- How does AI system design amplify model capabilities beyond the weights?
- Can scaffold-only modifications achieve lasting gains without updating the foundation model weights?
- How much does harness design separate from model intelligence affect benchmark scores?
- Do harness improvements eventually internalize into core model behavior over time?
- Does test-time scaffolding close capability gaps without retraining models?
- Can a harness design be optimized independently of the model inside it?
- How do evolved harness edits generalize across different benchmark domains?
- Do task-level outcomes provide sufficient supervision for harness evolution?
- Can harness edits distill reusable strategies or mostly memorize task-specific fixes?
- Can harness edits trained on one batch transfer to new tasks?
- Can harness evolution gains be distinguished from test-time search improvements on matched budgets?
- Can harness edits overfit to training tasks while appearing to improve performance?
- Do shell-state handling and answer aggregation fixes transfer across different models unchanged?
- How much token efficiency can harness-level intervention achieve versus training approaches?
- Would frontier closed models show the same harness indifference?
- Can individual harness components improve accuracy on specific benchmarks?
- Do harness-driven accuracy gains appear on other benchmarks?
- Does more inference compute help close gaps between different training regimes?
- How does distributed inference enable models to run on weak hardware?
- What efficiency gains could prevent an inference compute bottleneck?
- Does test-time compute substitute for parameter scaling at stable rates?
- How do architectural choices reduce inference cost independent of training compute?
- How much of Occamy's result comes from training versus the base model?
- Does teacher-style refinement of training data transfer equally to all student model distributions?
- Why is the fast non-parametric loop vulnerable to overfitting differently than model weights?
- What makes frozen model reasoning different from weight-based parameter updates?
- What distinguishes fast non-parametric loops from slow parametric weight updates?
- How much do cross-model improvements compare to the original performance gains on the training benchmark?
- Can split-softmax or other inference methods fully prevent instruction drift?
- Can corrected simulators replace real execution at inference time too?
- Can evolved algorithms transfer learning strategies across different datasets and tasks?
- How does learnability at the observer's current state prevent novelty from breaking model reasoning?
- Do tool-use capabilities actually help models recover from errors in reasoning?
- What happens when inference discovery occurs during live use rather than reflection?
- Does distilling experiment outcomes into reusable insights improve hypothesis quality over ranking alone?
- Does Amdahl's law or partial substitutability better model research task automation?
- How does model scale affect the crossover point between base and post-trained performance?
- Can task type alone predict which model size is sufficient?
- Do smaller models match Chinchilla-optimal performance when trained much longer?
- Do all models of equivalent quality see the same inference request volume?
- What happens to the optimal model size as projected inference volume grows?
Related concepts in this collection 3
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
-
Can non-reasoning models catch up with more compute?
Explores whether inference-time compute budget can close the performance gap between standard models and those trained for reasoning, and what training mechanisms might enable this.
contrast: extra inference compute leaves the gap open there, while a builder-written harness closes much of it here on Theory-of-Mind tasks
-
How do knowledge injection methods trade off flexibility and cost?
When and how should domain knowledge enter an AI system? This explores the speed, training cost, and adaptability trade-offs across four injection paradigms, and when each approach suits different deployment constraints.
harness is a no-training, no-parameter-update mechanism nearest prompt optimization, but includes code and routing the taxonomy does not describe
-
Does RL post-training create reasoning or just deploy it?
Investigates whether reasoning capability emerges during RL fine-tuning or already exists in base models. Matters because it reshapes how we build and optimize reasoning systems.
shared move of locating part of the gap in deployment rather than capability, by a different mechanism
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- AI4AI at Test-Time: Strong-to-Weak Capability Transfer via Harnesses
- Scaling Laws for Agent Harnesses via Effective Feedback Compute
- Evaluating Theory of Mind in Reasoning Models: Robustness over Reasoning
- Sharpening Tax in Post-Training
- The Invisible Leash: Why RLVR May Not Escape Its Origin
- Do Theory of Mind Benchmarks Need Explicit Human-like Reasoning in Language Models?
- The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexity
- Does Thinking More always Help? Understanding Test-Time Scaling in Reasoning Models
Original note title
a stronger builder model can transfer capability to a weaker target model at test time through a harness with no parameter updates