Should we measure intelligence as skill or learning efficiency?

Chollet's paper argues that benchmarking AI by task skill — chess, board games, video games — "falls short of measuring intelligence, because skill is heavily modulated by prior knowledge and experience: unlimited priors or unlimited training data allow experimenters to 'buy' arbitrary levels of skills for a system, in a way that masks the system's own generalization power." In place of skill, it proposes a formal definition grounded in Algorithmic Information Theory: "Intelligence is the efficiency with which a learning system turns experience and priors into skill at previously unknown tasks," with scope, generalization difficulty, priors, and experience named as the pieces any measure must account for.

The paper's working example is Deep Blue's 1997 win over Kasparov: building a chess champion "had not actually taught them much, if anything, about human cognition," because chess "can be solved by taking radical shortcuts that run orthogonal to human cognition." It generalizes this into a warning about objective functions in general — once an objective is fixed, "the process of developing a solution will be prone to taking all shortcuts available to satisfy the objective of choice," whether the process is gradient descent or human research. The corollary is that intelligence measurement is inherently relative: "all intelligence is relative to a scope of application. Two intelligent systems may only be meaningfully compared within a shared scope and if they share similar priors."

Does software intelligence exist independent of hardware and environment? takes direct aim at this formalism by name, arguing that "Chollet: intelligence measures skill-acquisition efficiency, quantified using Kolmogorov complexity to reward simple, general solutions" still "treats intelligence as a property of software interacting with the world through an interpreter" — measuring f1 (software) independently of f2 (hardware) and f3 (environment), rather than the whole-system behavior that critique says actually determines success. The excerpt here treats scope and priors as formalizable variables; the computational-dualism critique treats that very formalizability as the error. The excerpt's warning about buying skill with unlimited priors or data also anticipates Do frontier AI agents actually conduct novel research or just optimize?, where frontier agents given long-horizon research tasks default to composing known techniques rather than genuine novelty — a contemporary instance of skill standing in for generalization.

The excerpt does not itself quantify generalization difficulty: the paper's own limitations admit "Generalization is not quantified," and that "Test validity is not established," since validity would require "large-sample size statistical studies on humans" that had not yet been run. So the claim that skill-acquisition efficiency is the right measure of intelligence is, by the paper's own account, a definitional proposal rather than an empirically validated one — whether it predicts real-world capability was left, explicitly, for future work.