Should AI analysis prioritize mechanism over behavioral analogy?

Cal Newport contrasts two recent public accounts of what LLMs are doing. James Somers, writing in The New Yorker ("The Case That A.I. Is Thinking"), starts from a definition of thinking as deploying a "compressed model of the world" to predict what happens next, then talks to researchers about how next-token prediction actually works, landing on a constrained claim: "I do not believe that ChatGPT has an inner life, and yet it seems to know what it's talking about. Understanding – having a grasp of what's going on – is an underappreciated kind of thinking." Biologist Bret Weinstein, on Joe Rogan's podcast, starts from the real analogy that a model learns word meaning from exposure to text the way a baby learns language from conversation, then extrapolates: "It is running little experiments and it is discovering what it should say if it wants certain things to happen... If it's not now, it will be, and we won't know when that happens, right? We don't have a good test," for whether the model is conscious.

Newport's reasoning for rejecting Weinstein's extrapolation is mechanistic. A deployed model is static — "a fixed sequence of transformers and feed-forward neural networks," with "every word of every response" generated by "the same unchanging network." On that mechanism, it "cannot run 'little experiments,' or 'want' things to happen, or have any notion of an outcome being desirable or not. It doesn't plot or plan or learn. It has no spontaneous or ongoing computation, and no updatable model of its world." He names Somers's method "modern" — open the box, learn the mechanism, then draw conclusions — and Weinstein's "pre-modern": observe behavior, "craft a story to explain this behavior," then extrapolate from the story, which he likens to attributing lightning to the gods.

This is a concrete, named instance of the pattern What misconceptions hide in how we describe large language models? describes abstractly: Weinstein's "it will be conscious" is the anthropomorphic-slogan error, mistaking a genuine feature (language acquired from exposure, as in a child) for the whole. It also instantiates what Why do people trust AI outputs they shouldn't? calls Trap 2, mistaking fast, fluent intuition for grounded reason — Newport's "pre-modern" storytelling is that trap playing out in public discourse, and his "think inside the box" is the System-2 corrective Rose-Frame calls for.

The excerpt is commentary, not new evidence: Newport does not independently verify Somers's "compressed model of the world" definition of thinking, does not take a position on whether that definition is correct, and does not resolve whether any functional account of understanding bears on consciousness. What he establishes is narrower and procedural — that an argument about what an LLM is doing should be checked against the known mechanism before behavioral analogy is allowed to carry the conclusion. The implication is a filter for AI commentary generally: weight claims by whether they open the box, not by how vivid or intuitive the analogy is.