Can you really score how 'smart' an AI is without first agreeing what counts as fair knowledge to start with?
Why does every intelligence measure depend on shared scope and priors?
This explores why researchers argue that you can't say how intelligent a system is without first agreeing on what range of tasks counts (scope) and what built-in knowledge the system is allowed to start with (priors), and what goes wrong when those are left unstated.
This explores why an intelligence score only means something once you've agreed on two things: which tasks count, and what knowledge the system was allowed to bring with it. The clearest version of this argument comes from François Chollet, who says intelligence isn't the skill a system shows. It's how efficiently the system turns experience and starting knowledge into new skill Should we measure intelligence as skill or learning efficiency?. The reasoning is close to an accounting argument. Give a system unlimited training data or unlimited built-in knowledge and it can look skilled at almost anything without being able to adapt at all. A high score then tells you how much was bought, not how much was learned. To compare two systems fairly, you have to hold constant what each started with and define the set of tasks you're measuring over.
Chollet's ARC benchmark puts this into practice. It fixes the priors at a small set of 'core knowledge' that any human child has, such as objects, counting and basic geometry, and makes every task new, so neither memorization nor extra data can help Can a benchmark measure generalization without hiding training data?. Notice what this implies: ARC measures intelligence relative to human priors. A system that started from very different assumptions could be highly capable and still look weak on ARC, or the reverse. Shared priors don't make the measurement neutral. They make it comparable, which is a different thing.
The part you might not expect is that 'scope' covers more than the task list. It also includes where you draw the boundary of the system being measured. One line of work argues that defining intelligence as a property of software alone is a kind of Cartesian dualism, because real performance comes from software, hardware and environment working together Does software intelligence exist independent of hardware and environment?. There's a striking concrete case: a stronger model wrote inference-time scaffolding that nearly doubled a weaker model's Theory-of-Mind scores with no retraining. It did this mostly by moving shaky reasoning into deterministic code Can a stronger model lift a weaker one at test time without retraining?. So whose intelligence did the benchmark measure: the weak model's, the strong model's, or the combined system's? The number can't tell you unless the scope was fixed in advance. On a smaller scale, the same model's output changes with prompt wording and sampling Why does AI output change with every prompt and context?, so even the prompt is part of what's being measured.
Scope also decides what you're able to see. SOTOPIA measures social intelligence along seven separate dimensions, including goals, believability, keeping secrets, relationships and social rules. It also finds that humans reach their goals in about a third as many words as GPT-4 Can social intelligence be measured across seven dimensions?. If you measured only goal completion, that efficiency gap would be invisible. Mitchell and Krakauer go further: if LLMs understand through statistical correlation rather than causal models, then human-calibrated priors may be the wrong yardstick for them altogether Do large language models embody genuinely new modes of understanding?.
Put together, the corpus suggests this isn't a flaw to engineer away. It's what a measurement is. 'How intelligent?' is incomplete in the way 'how fast?' is incomplete without saying relative to what. The collection argues the case well, but it has less on practical ways to compare systems whose priors really do differ, such as humans and LLMs.
Sources 7 notes
Chollet argues intelligence should be measured as how efficiently a learning system converts experience and priors into capability on novel tasks, accounting for scope, generalization difficulty, priors, and experience. Task skill alone misleads because unlimited training data can artificially inflate performance without reflecting true learning capacity.
Chollet's ARC benchmark controls what test-takers can use (shared Core Knowledge priors) and what they haven't seen before (every task is novel) to measure generalization capacity. This design removes two levers—unlimited prior knowledge and unlimited training data—that allow systems to appear skilled without demonstrating real adaptability.
Influential AGI formalisms isolate intelligence in software independently of hardware and environment, but success depends on all three layers together. This mirrors Cartesian dualism—a fundamental error that makes isolated benchmarks inadequate measures of AGI.
A stronger model built inference-time harnesses that nearly doubled weaker model performance on Theory-of-Mind benchmarks without retraining, primarily by moving unstable reasoning into deterministic code and task-specific routing rather than encouraging extended reasoning.
AI outputs exhibit essential mutability—they vary with sampling, prompt wording, and audience interpretation. This is not a defect but a defining feature of tokens as media, making them fundamentally different from fixed commodities and resistant to traditional quality assurance.
Show all 7 sources
SOTOPIA framework operationalizes social intelligence across Goal, Believability, Knowledge, Secret, Relationship, Social Rules, and Financial dimensions. Humans produce 16.8 words per turn versus GPT-4's 45.5, revealing efficiency as a measurable capability in social interaction.
Mitchell and Krakauer argue that LLMs represent a distinct species of intelligence that operates through statistical correlations rather than causal mental models. They propose this warrants an expanded science of intelligence that moves beyond the binary debate of whether LLMs truly understand.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- On the Measure of Intelligence
- The Missing Layer of AGI: From Pattern Alchemy to Coordination Physics
- What the F*ck Is Artificial General Intelligence?
- The Debate Over Understanding in AI's Large Language Models
- AI4AI at Test-Time: Strong-to-Weak Capability Transfer via Harnesses
- Word Meanings in Transformer Language Models
- From Human to Machine Psychology: A Conceptual Framework for Understanding Well-Being in Large Language Models
- Beyond Hallucinations: The Illusion of Understanding in Large Language Models