SYNTHESIS NOTE
Topics›MechInterp›this note

Can we read a language model's unspoken thoughts?

Does the Jacobian lens successfully identify what a model is poised to verbalize, and do those representations function as a global workspace analogous to human conscious access?

Synthesis note · 2026-09-25 · sourced from MechInterp

The paper claims that language models show a functional split analogous to the one in human cognition, where only a small fraction of processing is "consciously accessible" for verbal report, deliberate control, and flexible reasoning. Using a new technique, the Jacobian lens, it identifies "the representations a model is poised to verbalize at any point in its processing" and calls them the J-space. Their contents "can be reported, deliberately summoned and held, used to carry the intermediate steps of silent reasoning, and passed as arguments to arbitrary downstream computations," while "automatic processing such as text parsing and routine inference proceeds without them."

Beyond the functional properties, the abstract lists three structural signatures that global workspace theory associates with conscious access. The J-space carries coherent content only in an intermediate band of layers, holds "on the order of tens of concepts at a time," and is broadcast by the model's weights more widely than other representations. The defining move is to carve out the verbalizable subset of internal state instead of reading all of it. The discussion adds the practical side: the lens readout is "a single matrix multiplication per layer, with the matrix computed once per model," needs no auxiliary training, and produces output a human can read directly. In the auditing case studies, strategic and situational assessments appeared in the J-space "even when not visible (or prior to being visible) in the model's output," and a misaligned disposition implanted by training left "a standing signature" at the start of responses whose surface behavior was unremarkable.

Against the neighbors, this is a representation-level readout in the same top-down spirit as Can high-level concepts replace circuit-level analysis in AI?, but it is defined by what the model can verbalize instead of by a predefined concept direction, and it claims a functional partition of the model's state. It shares a question with Can language models detect their own internal anomalies?, namely which internal states a model can access and report; the excerpt does not say the J-space accounts for those introspection results. It also qualifies Can models reason without generating visible thinking tokens?: on the paper's account, silent reasoning steps are carried by representations the model is poised to verbalize, so reasoning that is silent in the output is not necessarily unreadable inside.

The excerpt is silent on nearly everything a reader would need to weigh the evidence. It names no models, layer ranges, or tasks, gives no measurements behind the "tens of concepts" figure or the functional properties, and reports no detection rates, false positives, or comparison against other monitors for the auditing and model-organism results. "Global workspace" is used as a functional analogy, and the excerpt makes no claim about experience. The alignment case is explicitly conditional: "If a model's strategic deliberation routes through the J-space," then inspecting it at decision positions will reveal that deliberation. The results only "suggest" the lens "could be highly useful" for monitoring. The defensible reading is a cheap first-pass flag for transcript review, not a verified monitor, and deliberation that does not route through the J-space would fall outside it.

Inquiring lines that read this note 2

This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.

What mechanisms preserve shared understanding in evolving conversations? Does encoded knowledge in language models actually influence their outputs?

Related concepts in this collection 4

This note in its neighbourhood — explore the map, then jump to a related concept in the list below.

Concept map
13 direct connections · 113 in 2-hop network ·medium cluster Open in graph ↗

Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph

your link semantically near linked from elsewhere

Related papers in this collection 8

Papers most semantically related to this note, ranked by cosine similarity in the embedding space.

Original note title

the representations a language model is poised to verbalize form a global workspace — the Jacobian lens reads it out as a window into unspoken thinking