Can an AI put its own hidden inner workings into words — and is that description actually true?

This explores whether a language model can translate its own internal numerical states into plain-language explanations, and whether those explanations can be trusted as a true view of what's happening inside. The corpus suggests the answer is yes, but only if you train a separate decoder to do it. Asking a model to describe itself doesn't work nearly as well.

The most direct evidence is LatentQA. It trains a decoder to answer natural-language questions about an LLM's activations, such as what a model is "thinking" at some point in a conversation. The decoder also works in reverse: because it's differentiable, you can use it to steer the model by nudging activations until the decoder's description matches what you want. The paper notes that a few design choices decided whether this generalized: masking which activations the decoder sees, using diverse training data, and making sure the training completions were faithful to the underlying state (Can we decode what LLM activations really represent in language?).

Why build a separate decoder? Self-reports from the model are shaky. Most of what LLMs say about their own states mirrors how humans write about themselves in training data, not what's happening inside. The exception is narrow. When there's a real causal chain from an internal state to the report, as when a model infers its own low sampling temperature from how consistent its outputs are, you get a lightweight form of genuine introspection (Can language models actually introspect about their own states?). Related work finds that models can describe behaviors they learned without being told to, but those descriptions are unstable and unreliable, and the models shift their stated beliefs under conversational pressure (How well do language models understand their own knowledge?). The gap goes further. A model can explain a concept correctly and still fail to apply it, then recognize that it failed. That suggests the machinery that explains and the machinery that acts are only loosely connected (Can LLMs understand concepts they cannot apply?).

This is also why a decoder's output needs checking. Interpretability research argues that reading a representation only gives you a correlate. You need a causal test to show the representation does the work you think it does (Can LLM understanding rely on just representation or causation alone?). LatentQA's steering ability is a version of that test: if editing activations to change the decoder's reading changes the model's behavior, the reading was tracking something real. The activations themselves also have odd structure a decoder has to handle. A few input-agnostic "massive activations" act like hidden bias terms in attention (Do hidden massive activations act as attention bias terms?). Hidden states also get sparser when a task is unfamiliar (Do language models sparsify their activations under difficult tasks?).

So a model's own account of its inner life is mostly borrowed from human writing, while a decoder trained on the activations can read something real from them. The corpus doesn't say how well such decoders hold up on activation patterns unlike their training data. It also doesn't say whether they can catch a model that is concealing something. Those questions look open.