Researchers can peek inside an AI to see what it 'knows' — but how do they tell real insight from coincidence?

This explores the toolkit researchers use to look inside a language model and work out what it is representing, and how much to trust what those tools show. The corpus's central lesson is that no single method is enough. Representational analysis, meaning finding where a concept seems to live inside the model, only turns up correlates. Causal analysis, meaning nudging or knocking out a part and watching what changes, shows effects but not what the part is for. Real mechanistic understanding comes from pairing them: locate a candidate representation, then verify by intervention that the model actually uses it (Can LLM understanding rely on just representation or causation alone?). Cognitive science has run this same play for decades, and one note argues its behavioral probes, causal interventions and representational analysis carry over to LLMs, organized by Marr's levels of analysis (what a system computes, by what procedure, in what hardware) (Can cognitive science methods unlock how LLMs actually work?).

When the methods work, they surface different kinds of internal structure. Mechanistic interpretability evidence points to three tiers. First, concepts stored as directions in the model's internal space. Second, connected facts about the world. Third, compact circuits that implement a principled algorithm. The higher tiers sit alongside lower-tier heuristics instead of replacing them, so a model is a patchwork, not one uniform kind of understanding (Do language models understand in fundamentally different ways?). Sparse autoencoders, which split a model's activity into individually interpretable features, found a mechanism that tracks whether the model recognizes an entity. That mechanism causally steers whether the model hallucinates or refuses, and it survives from base models into chat-tuned ones (Do models know what they don't know?). Another study traced a two-stage circuit that lets a model notice when someone has injected a steering vector into its activations. The circuit emerges from DPO training, and safety training can suppress it, dropping detection from 63.8% to 10.8% (How do language models detect injected steering vectors internally?).

You can also watch representations change as a computation moves through the layers. The deep-thinking ratio counts how many tokens have their predictions substantially revised in later layers. That turns out to track genuine reasoning effort and correlates with accuracy on hard math and science benchmarks (Can we measure how deeply a model actually reasons?). Looped transformers show the same idea from the other side. Each loop settles into a stable, repeating trajectory that mirrors the stages of ordinary feedforward inference, so extra depth mostly re-runs existing computation rather than inventing new kinds (How do looped language models actually improve reasoning in depth?).

The reason for all this effort is that outputs alone can hide the internal picture. Models with identical accuracy can have radically different internals, including fractured representations where every needed feature is linearly decodable but the organization is broken. That leaves the model fragile under perturbation and distribution shift in ways standard benchmarks miss (Can models be smart without organized internal structure?, What really happens inside a language model?). Asking the model to describe itself is a poor shortcut. Most self-reports echo human training text, and only where a causal chain links an internal state to the report does lightweight introspection occur (Can language models actually introspect about their own states?). Natural-language explanations also widen what humans can be told, and they open the door to plausible-sounding explanations that aren't faithful to the model (Can natural language explanations redefine what interpretability means?). That is why methods like RecExplainer train an LLM to mimic a recommender's internal states, so its explanations stay tied to the real model (Can LLMs explain recommenders by mimicking their internal states?).