When you test an AI, how much of its score comes from skill it actually learned versus knowledge it was just handed beforehand?

This explores where to draw the line between what a system already brings to a test (its priors) and what it learns to do (its skills), and why that line decides whether a high score means anything. The clearest framing in the collection comes from François Chollet, who argues that intelligence isn't skill at all. It is how efficiently a system turns its priors plus its experience into new skill on tasks it hasn't seen Should we measure intelligence as skill or learning efficiency?. On this view, a skill is the output you can observe, such as solving a puzzle or passing an exam. Priors are what the system had before it started, and experience is the practice it got along the way. If you ignore priors and experience, you can buy skill without buying intelligence: give a system enough training data, or build the answer in, and its scores go up while its ability to learn stays where it was.

That definition is tidy for humans, whose priors are mostly innate. For language models it gets messy, because pretraining is both experience and prior at once. Several notes suggest that much of what looks like a newly taught skill is really a prior being drawn out. Five separate methods, including RL, critique fine-tuning, decoding changes and feature steering, all appear to unlock reasoning that was already present in the base model. Post-training selects rather than creates Do base models already contain hidden reasoning ability?. The same pattern shows up in theory-of-mind tests: reasoning models score better mainly because their answers hold steady when the prompt is reworded, not because they gained a new capacity Does reasoning training improve theory of mind or just stability?. So a jump in benchmark scores after training may tell you more about how well a prior was drawn out than about how much skill was acquired.

The collection also hints at what these priors are made of. A study tracing reasoning back through millions of pretraining documents found that reasoning draws on broad, reusable procedural knowledge: general how-to patterns gathered from many sources. Factual recall, by contrast, depends on memorizing specific documents Does procedural knowledge drive reasoning more than factual retrieval?. A cipher study splits chain-of-thought performance into three ingredients that operate at once: how likely the answer is as text, how often similar material appeared in training, and real but error-prone step-by-step reasoning What three separate factors drive chain-of-thought performance?. That gives a practical test. If a score rises and falls with how common the answer is in training data, you are mostly measuring priors. If it holds up on rare or unfamiliar versions of the task, something closer to skill acquisition is going on.

There is a twist. You might expect real reasoning to ignore content, applying the same logic whether a puzzle is about beer and ages or abstract symbols. But humans don't reason that way either. People and LLMs succeed and fail on the same content-sensitive reasoning tasks Do language models fail reasoning tests that humans pass?. Relying on priors isn't cheating. All reasoners do it. The question is whether the test separates the help a prior gives from the learning that happens on top of it. Even the word "skill" is slippery: in agent systems, an added "skill" mostly steadies how the agent acts rather than giving it missing facts Do skills teach procedures or inject missing facts?.

What you might not have expected: the prior-versus-skill problem isn't limited to AI tests. Studies of people using AI hit the same trap. Telemetry shows expertise being used but can't show whether expertise is being built Can we measure whether AI erodes independent skill?. Users also mistake the fluency of AI output for their own competence Does processing ease mislead users about their own competence?. Whether the subject is a model or a person, a polished performance can hide whether anything was learned. The collection has a lot on how to separate the two in models but little on how to measure priors directly in humans working with AI.