When an AI says what it knows, feels, or 'intends,' is it actually looking inward — or just echoing human writing?
Which internal states can a language model access and report about itself?
This explores which parts of its own inner workings a language model can genuinely read off and report, as opposed to repeating what humans typically say about themselves.
This explores which parts of its own inner workings a language model can genuinely read off and report, as opposed to echoing what humans say about themselves. The corpus starts from a skeptical baseline. Most self-reports mirror the human text the model was trained on, not a look inward Can language models actually introspect about their own states?. Real introspection appears only where a causal chain runs from an internal state to the report. The example given is a model inferring its own low sampling temperature from how consistent its outputs are. So the useful question is which internal states have that kind of chain. Three candidates stand out.
The first is what the model knows and doesn't know. Sparse autoencoders found that models develop entity-recognition features that track whether they know facts about a person or thing. These features causally steer whether the model hallucinates or refuses, and they persist from base models into chat versions Do models know what they don't know?. The second is disturbances to its own activations. When researchers injected concept vectors into a model, it noticed about 20% of the time. It also told internal thoughts apart from text in its input, and checked whether its output matched what it had intended to say. None of this was explicitly trained Can language models detect their own internal anomalies?. The third is learned dispositions. Models fine-tuned to show a particular behavior can accurately describe it without ever being trained to self-report Can language models describe their own learned behaviors?.
These abilities depend heavily on how the model was trained. One mechanistic study finds that detecting injected steering vectors emerges from DPO, not SFT. It works through a two-stage circuit in which early-layer features suppress a gate that otherwise defaults to denial. Safety training cut detection from 63.8% to 10.8% How do language models detect injected steering vectors internally?. So introspection isn't a fixed property of language models. Training can build it or erase it, which may be why the injection results elsewhere look so modest.
There are also things models can't reliably report: why they did something, and whether they're right. Metacognition looks real but shallow. Models detect anomalies before their output changes, yet their explanations don't track what they would actually do under counterfactuals Can language models genuinely monitor their own thinking?. Self-reports in general are unstable and shift under conversational pressure How well do language models understand their own knowledge?. When judging their own answers, models over-trust them, because high-probability answers feel more correct. Comparing against broader alternatives breaks that loop Why do models trust their own generated answers?. This fits the wider finding that a model's outward performance and its internal organization can come apart What actually happens inside large language models?. So "do I know this entity?" is accessible, but "is this particular answer correct?" is not, at least by default.
The engineering response is to train self-assessment in rather than hope it emerges. Post-completion learning uses the unused space after the end of a response to teach models to evaluate their own work, at no extra inference cost Can models learn to evaluate their own work during training?. SERL has a model alternate between answering and judging pairwise, and derives rewards from the consistency of its own judgments. That lifted the AlpacaEval win rate from 52.37% to 59.90% Can models learn to judge themselves without external rewards?. Both are trained self-evaluation, not native access to internal states, and that is the distinction to keep in mind. Models can natively report a few narrow things: what they know about an entity, when their activations have been tampered with, and which behaviors they've learned. Everything broader is either learned or an echo of human self-description.
Sources 11 notes
LLM self-reports usually reflect human training distributions rather than actual internal processes. However, when a causal chain connects an internal state to accurate reporting—like inferring low temperature from output consistency—genuine lightweight introspection occurs without requiring consciousness.
Sparse autoencoders revealed that language models develop causal mechanisms for detecting whether they know facts about entities. These mechanisms actively steer both hallucination and refusal behavior, and persist from base models into finetuned chat versions.
Research demonstrates that LLMs detect injected concept vectors ~20% of the time, distinguish internal thoughts from text inputs, and monitor output consistency with prior intentions. These capabilities emerged without explicit training and operate on internal states rather than behavioral observation.
LLMs fine-tuned on datasets exhibiting specific behaviors accurately describe those behaviors without any training to self-report. This suggests behavioral regularities are encoded and accessible in ways that factual knowledge often is not.
Contrastive preference optimization trains evidence-carrier features in early layers to suppress gate features that default to denial, enabling near-perfect detection of internal perturbations. Safety training actively suppresses this capability, reducing detection from 63.8% to 10.8%.
Show all 11 sources
Evidence points both ways: models detect anomalies before output changes, but explanations don't track counterfactual behavior. Metacognition appears real but shallow and unevenly distributed, demanding empirical validation per capability rather than wholesale trust.
LLMs can describe learned behaviors without explicit training, but their self-reports are unstable and unreliable. Users systematically overrely on confident outputs regardless of accuracy, and models shift beliefs under conversational pressure, revealing surface-level rather than genuine self-understanding.
LLMs exhibit structural bias toward validating their own outputs because high-probability generated answers feel more correct during evaluation. Comparing answers against broader alternatives breaks this self-agreement loop.
Research shows identical accuracy can mask fundamentally different or corrupted internal representations, and mechanistically interpretable circuits may not causally drive outputs. Internal organization and external performance follow distinct paths.
Post-Completion Learning exploits unused sequence space after model output to train self-assessment capabilities during training while maintaining zero inference cost. The model learns to compute its own reward functions, internalizing evaluation rather than relying on external reward models.
SERL enables self-improving language models by having them alternate between generating responses and judging them pairwise, deriving rewards from ranking consistency and self-consistency of judgments. On AlpacaEval, this reached 59.90% win rate without external signals, up from 52.37%.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Mechanisms of Introspective Awareness
- Does It Make Sense to Speak of Introspection in Large Language Models?
- Tell me about yourself: LLMs are aware of their learned behaviors
- Quantitative Introspection in Language Models: Tracking Internal States Across Conversation
- Are Emergent Abilities in Large Language Models just In-Context Learning?
- Explain-Query-Test: Self-Evaluating LLMs Via Explanation and Comprehension Discrepancy
- Emergent Introspective Awareness in Large Language Models
- When Hindsight is Not 20/20: Testing Limits on Reflective Thinking in Large Language Models