A strong score on brand-new puzzle tasks sounds impressive — but does it mean an AI will actually be useful at real work?
Does ARC performance on novel tasks predict real-world usefulness?
This explores whether doing well on ARC, a puzzle benchmark built so that every task is new, tells you much about whether an AI system will be useful on real work. The library doesn't test that link directly, but it has a lot to say about what benchmarks measure and what they miss.
This explores whether doing well on ARC, a puzzle benchmark built so that every task is new, tells you much about whether an AI system will be useful on real work. The short answer from the collection: no paper here measures that link. What the collection does show is why ARC was built the way it was, and why the gap between benchmark scores and usefulness keeps appearing.
ARC is designed to defeat the usual ways systems look smarter than they are. It allows only basic 'core knowledge' (objects, counting, simple geometry) and makes every task one the system has never seen. Without huge amounts of prior knowledge or training data to lean on, a high score has to come from adapting on the spot Can a benchmark measure generalization without hiding training data?. That design makes more sense once you see how easily other measures get fooled. Models trained to copy ChatGPT fool human judges with a confident, fluent style while gaining nothing in accuracy or in handling new tasks Can imitating ChatGPT fool evaluators into thinking models improved?. Instruction tuning with empty or deliberately wrong instructions does almost as well as tuning with correct ones, which suggests models mostly learn what an answer should look like Does instruction tuning teach task understanding or output format?. Chain-of-thought examples with broken logic work nearly as well as valid ones Does logical validity actually drive chain-of-thought gains?. ARC is in effect an answer to all of this: a test where looking right doesn't earn points.
The flip side is that much of what makes models look capable is familiarity rather than adaptability. Chain-of-thought reasoning breaks down in predictable ways once the task, length or format drifts away from the training data Does chain-of-thought reasoning actually generalize beyond training data?. The 'zero-shot' performance of image-and-text models tracks how often a concept appeared in pretraining, and each steady improvement takes exponentially more data Does multimodal zero-shot performance actually generalize or interpolate?. So ARC isolates a real and scarce ability. But real-world usefulness often rewards that same familiarity. Most practical work looks like something the model has seen before, so a system can be weak at novel puzzles and still be very useful on ordinary tasks.
The less obvious lesson is that real-world usefulness often turns on things ARC doesn't touch at all. Vision-only GUI agents fail on screens partly because they're asked to work out what each icon means and decide what to do in the same step. Splitting those jobs apart fixes much of the problem without making the model any smarter Why do vision-only GUI agents struggle with screen interpretation?. Automatically tuning agent harnesses cut token use by nearly half with no change to the model, gains the authors call orthogonal to model improvements Can agent harnesses be automatically optimized across many environments?. When researchers want evidence that gains carry over, they check held-out tasks directly, as AIDE2 did with weather forecasting outside its training set Do AIDE2's improvements transfer to unseen tasks?. They don't rely on a single test of general intelligence.
The takeaway: an ARC score is a fairly honest sign of one thing, adapting to the genuinely new, and the collection explains why that is rare and hard to fake. It is a poor stand-in for usefulness, because usefulness also depends on familiarity with common tasks and on the surrounding setup (harness, interface, tooling), which can matter as much as the model. If you want to know whether a system will help with your work, the collection points toward testing it on held-out versions of that work.
Sources 9 notes
Chollet's ARC benchmark controls what test-takers can use (shared Core Knowledge priors) and what they haven't seen before (every task is novel) to measure generalization capacity. This design removes two levers—unlimited prior knowledge and unlimited training data—that allow systems to appear skilled without demonstrating real adaptability.
Imitation models fool human evaluators by mimicking ChatGPT's confident, fluent style while failing to improve factuality or generalization on novel tasks. The ceiling is set by base model capability, not fine-tuning method—better fundamentals, not shortcuts, drive real improvement.
Models trained on semantically empty or deliberately incorrect instructions achieve comparable performance to those trained on full correct instructions, achieving 43% vs random baseline 42.6%. The semantic content of instructions appears largely irrelevant; what transfers is knowledge of the output space.
Illogical chain-of-thought exemplars matched valid CoT performance on BIG-Bench Hard, showing that structural properties—not logical validity—drive the gains. The model learns the form of reasoning, not genuine inference.
DataAlchemy experiments show CoT fails systematically under distributional shifts in task, length, and format. Models produce fluent but logically inconsistent reasoning — imitating reasoning form without valid underlying logic.
Show all 9 sources
Across 34 models and 5 datasets, multimodal models require exponentially more pretraining data for linear performance gains on downstream tasks. Performance correlates with how often test concepts appeared during pretraining, not genuine generalization ability.
OmniParser demonstrates that GPT-4V fails when forced to simultaneously identify icon meanings and predict actions from raw screenshots. Pre-parsing screenshots into structured semantic elements with descriptions lets the model focus solely on action prediction, removing the composite-task bottleneck.
Cross-environment harness optimization yielded four mechanisms (action execution, context compaction, observation handling, delegated reading) that reduced token traffic by 44.7–49.0% while maintaining comparable performance on a 51-task benchmark, suggesting harness-level gains are orthogonal to model improvements.
The paper reports that AIDE2's improvements transfer to four held-out benchmarks spanning machine learning, algorithm engineering, and physics-based weather forecasting—the last being outside the selection task distribution. This demonstrates transferable gains beyond overfitting to the selection set.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Is Chain-of-Thought Reasoning of LLMs a Mirage? A Data Distribution Lens
- When More is Less: Understanding Chain-of-Thought Length in LLMs
- Measuring Faithfulness in Chain-of-Thought Reasoning
- CoT is Not True Reasoning, It Is Just a Tight Constraint to Imitate: A Theory Perspective
- AI Agents That Matter
- Automated Alignment Researchers: Using large language models to scale scalable oversight
- MOMENTS: A Comprehensive Multimodal Benchmark for Theory of Mind
- From Entropy to Epiplexity: Rethinking Information for Computationally Bounded Intelligence