Counting an AI's parameters is like measuring what a student could have memorized, not what they actually did.

This explores why counting parameters is a poor proxy for a model's true simplicity — and the corpus points at a clean answer: parameter count measures the *capacity you allocated*, not the *content the model learned*. The two come apart badly. The most direct treatment is requential coding, which measures learned simplicity by coding only where a student disagrees with a teacher Can we measure what a model truly learned?. By ignoring everything the model got right for free and paying bits only for genuine disagreement, it produces compressed sizes *independent of parameter count and data entropy* — orders of magnitude shorter than prequential methods. That independence is the whole point: if your measure moves when you change parameter count without changing what was learned, it's measuring the wrong thing.

The reason parameter-based measures mislead is that a model can carry all the right surface behavior on top of broken internal structure. Two models with identical performance can have fundamentally different internal representations Can models be smart without organized internal structure? — one can be linearly decodable and accurate yet internally fractured, vulnerable to perturbation in ways no standard metric sees. If accuracy and decodability can hide this, so can parameter count. Simplicity is about how *compressed and organized* the learned content is, not how many weights hold it.

There's a deeper framing here: learning *is* compression. Optimal language-model training can be derived directly from a lossless compression objective Does optimal language model learning maximize data compression?, and text-trained models act as general-purpose compressors that beat specialized tools like PNG and FLAC by adapting in-context rather than by storing more Can text-trained models compress images better than specialized tools?. If capability comes from compression quality, then the honest measure of what a model learned is *description length of the learned content* — how short a code the model lets you write — not the size of the container. Parameter count is the container.

Capacity and content diverging shows up architecturally too. Depth beats width for sub-billion-parameter models Does depth matter more than width for tiny language models?: the same parameter budget yields more capability when composed through layers than spread across width — so even *identical* parameter counts encode different amounts of learned structure depending on arrangement. And compression itself isn't neutral: LLMs compress more aggressively than humans, capturing broad category structure while shedding the fine distinctions humans keep Do LLMs compress concepts more aggressively than humans do?. So 'simplicity' has a direction — you can compress your way into losing exactly the nuance that mattered.

The thing you didn't know you wanted to know: the right measure of model simplicity is relational, not absolute. Requential coding only works because it measures a *student against a teacher* — disagreement, not raw size. Simplicity isn't a property you can read off a parameter file any more than the difficulty of a proof can be read off its page count Can autoformalization work on individual statements alone?. It's a measure of how little new information the model needed to add on top of what was already predictable — and parameters can't see that, because they were bought before any learning happened.