Can we read a language model's unspoken thoughts?
Does the Jacobian lens successfully identify what a model is poised to verbalize, and do those representations function as a global workspace analogous to human conscious access?
The paper claims that language models show a functional split analogous to the one in human cognition, where only a small fraction of processing is "consciously accessible" for verbal report, deliberate control, and flexible reasoning. Using a new technique, the Jacobian lens, it identifies "the representations a model is poised to verbalize at any point in its processing" and calls them the J-space. Their contents "can be reported, deliberately summoned and held, used to carry the intermediate steps of silent reasoning, and passed as arguments to arbitrary downstream computations," while "automatic processing such as text parsing and routine inference proceeds without them."
Beyond the functional properties, the abstract lists three structural signatures that global workspace theory associates with conscious access. The J-space carries coherent content only in an intermediate band of layers, holds "on the order of tens of concepts at a time," and is broadcast by the model's weights more widely than other representations. The defining move is to carve out the verbalizable subset of internal state instead of reading all of it. The discussion adds the practical side: the lens readout is "a single matrix multiplication per layer, with the matrix computed once per model," needs no auxiliary training, and produces output a human can read directly. In the auditing case studies, strategic and situational assessments appeared in the J-space "even when not visible (or prior to being visible) in the model's output," and a misaligned disposition implanted by training left "a standing signature" at the start of responses whose surface behavior was unremarkable.
Against the neighbors, this is a representation-level readout in the same top-down spirit as Can high-level concepts replace circuit-level analysis in AI?, but it is defined by what the model can verbalize instead of by a predefined concept direction, and it claims a functional partition of the model's state. It shares a question with Can language models detect their own internal anomalies?, namely which internal states a model can access and report; the excerpt does not say the J-space accounts for those introspection results. It also qualifies Can models reason without generating visible thinking tokens?: on the paper's account, silent reasoning steps are carried by representations the model is poised to verbalize, so reasoning that is silent in the output is not necessarily unreadable inside.
The excerpt is silent on nearly everything a reader would need to weigh the evidence. It names no models, layer ranges, or tasks, gives no measurements behind the "tens of concepts" figure or the functional properties, and reports no detection rates, false positives, or comparison against other monitors for the auditing and model-organism results. "Global workspace" is used as a functional analogy, and the excerpt makes no claim about experience. The alignment case is explicitly conditional: "If a model's strategic deliberation routes through the J-space," then inspecting it at decision positions will reveal that deliberation. The results only "suggest" the lens "could be highly useful" for monitoring. The defensible reading is a cheap first-pass flag for transcript review, not a verified monitor, and deliberation that does not route through the J-space would fall outside it.
Inquiring lines that read this note 2
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
What mechanisms preserve shared understanding in evolving conversations? Does encoded knowledge in language models actually influence their outputs?Related concepts in this collection 4
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
-
Can high-level concepts replace circuit-level analysis in AI?
Instead of reverse-engineering individual circuits, can we study AI reasoning by treating concepts as directions in activation space? This matters because circuit analysis hits practical limits at scale.
another representation-level, top-down readout; the J-space is defined by verbalizability rather than by a concept direction chosen in advance
-
Can language models detect their own internal anomalies?
Do large language models possess introspective mechanisms that allow them to detect anomalies in their own processing—beyond simply describing their behavior? The answer has implications for both AI transparency and deception.
both concern which internal states a model can report; the excerpt does not link the J-space to injected-concept detection
-
Can models reason without generating visible thinking tokens?
Explores whether intermediate reasoning must be verbalized as text tokens, or if models can think in hidden continuous space. Challenges a foundational assumption about how language models scale their reasoning capabilities.
treats reasoning without verbalized tokens as a scaling route; this paper says such silent steps pass through verbalizable representations
-
Can we trigger reasoning without explicit chain-of-thought prompts?
This research asks whether models possess latent reasoning capabilities that can be activated through direct feature steering, independent of chain-of-thought instructions. Understanding this matters for making reasoning more efficient and controllable.
another account of reasoning happening off the output channel; the excerpt does not compare the two
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- Verbalizable Representations Form a Global Workspace in Language Models
- Computational structuralism: Toward a formal theory of meaning in the age of digital intelligence
- Mechanistic Indicators of Understanding in Large Language Models
- Beyond Hallucinations: The Illusion of Understanding in Large Language Models
- Semantic Structure in Large Language Model Embeddings
- Farther the Shift, Sparser the Representation: Analyzing OOD Mechanisms in LLMs
- Generative Models as a Complex Systems Science: How can we make sense of large language model behavior?
- Beyond Language Modeling: An Exploration of Multimodal Pretraining
Original note title
the representations a language model is poised to verbalize form a global workspace — the Jacobian lens reads it out as a window into unspoken thinking