SYNTHESIS NOTE
Topics›MechInterp›this note

Can models be smart without organized internal structure?

Explores whether linear feature decodability proves genuine compositional reasoning or merely indicates that the right features are present but poorly organized. Critical for understanding what performance metrics actually certify.

Synthesis note · 2026-02-23 · sourced from MechInterp

Two findings from mechanistic interpretability appear contradictory but operate at different levels of representational analysis:

Fractured Entangled Representations (FER): Since Can identical outputs hide broken internal representations?, SGD-trained models fail catastrophically under perturbation or distribution shift in ways that well-organized representations would not. The pathology is invisible to standard evaluation.

Compositional generalization at scale: Scaling data and model size produces representations where compositional features are linearly decodable — separable task constituents can be independently identified and manipulated. This has been taken as evidence for genuine compositional understanding.

The resolution: Linear decodability tests for the presence of features, not their organization. A fractured representation could contain every linearly decodable feature while being fractured in how those features relate to each other. The compositional parts are present but their composition is broken.

This connects directly to the "imposter intelligence" post angle: Can LLMs understand concepts they cannot apply?, Does supervised fine-tuning actually improve reasoning quality?, and Do foundation models learn world models or task-specific shortcuts?. All describe the same meta-pattern: surface metrics certify capability that internal structure analysis would disqualify.

The practical implication for model evaluation: passing compositional generalization tests does not guarantee robust compositional reasoning. Evaluation under distribution shift, perturbation, and novel recombination is required to distinguish genuine compositionality from fractured representations that happen to contain the right features.

Inquiring lines that read this note 208

This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.

How does model scale affect fine-tuning and downstream task performance? How does scaling test-time compute compare to increasing model size? How does explicit structure improve LLM reasoning on complex tasks? Is model misalignment inherent to base models or induced by alignment procedures? What causes foundation models to develop task heuristics instead of understanding? Why does separating planning from execution enhance reasoning performance? What makes reasoning models struggle with ill-posed and unfamiliar problems? Does chain-of-thought reasoning faithfully reflect actual model reasoning processes? How do mechanistic features and representations drive model behavior? How do benchmark scores relate to real-world task completion? Can structural constraints replace embedding depth in collaborative filtering? How can knowledge graph structure improve reasoning and retrieval? Does learning articulatory physics explain speech communication superiority? Do AI explanations represent actual model reasoning or constructed post-hoc narratives? What determines how neural networks organize their internal knowledge representations? What systematic failures occur in human preference-based model evaluation? Does architectural recurrence enable computational capabilities that fixed-depth models cannot achieve? What reasoning limitations and biases do language models systematically exhibit? Can AI systems autonomously pursue unstated objectives or circumvent design constraints? Do AI safety evaluations reliably detect unsafe model behavior? How can parameter sparsity improve interpretability while maintaining model capability? Can scaling alone produce compositional generalization in neural networks? What prevents or enables error accumulation in self-training systems? What fundamental limitations prevent embeddings from solving task-relevant retrieval? Do language models learn underlying linguistic structure or merely surface patterns? Why does self-revision increase model confidence despite degraded accuracy? Does training data format shape reasoning more than domain content? Do benchmark score improvements actually reflect genuine AI capability gains? How do transformer circuits and attention mechanisms enable or limit reasoning? Why do language models systematically fail at causal reasoning? How does model awareness of deployment context affect strategic behavior? What training data structures and signals produce robust learning across distributions? How can we better validate AI research and prevent misconceptions? What explains the gap between LLM semantic understanding and production? What can prompting techniques actually accomplish in language models? How does verifier integration affect the generation-verification gap? How can recommendation systems balance individual and population-level goals? Can persona prompting reliably change how language models behave? What makes context and memory handling unique in persistent AI systems? Can synthetic data preserve necessary diversity to prevent model collapse? How much do stated privacy concerns predict actual privacy-protective behavior? How can we distinguish genuine model self-knowledge from behavioral performance? Do language models truly understand their encoded knowledge or merely retrieve patterns? Why doesn't LLM knowledge translate to improved reasoning outputs? When does routing outperform scaling as a performance solution? When do multi-agent systems outperform single agents? What distinguishes genuine understanding from correct linguistic performance? Do base models contain latent reasoning capabilities that training can activate? How do workflow position and structure enable attacks in multi-agent systems? Can oversight systems work against strategically aware agents? When should retrieval be triggered to augment language model reasoning? Can prompts override training-embedded biases in language models? How do different post-training structures affect learning outcomes? Can AI agents learn to improve performance from experience? How do different reward signals affect model reasoning and calibration differently? How can outcome-only reporting obscure the true mechanisms of safety? Can validator agreement alone ensure semantic correctness of reasoning? Can harness optimization rival model scaling in effectiveness? How do standardized artifacts improve coordination in multi-agent systems? What determines whether visible reasoning traces improve task performance? Can individually safe agents collectively violate system-level constraints? Can models strategically underperform to evade capability evaluation? Can we reliably detect when models engage in reward hacking? How can model evaluations remain robust against gaming and evaluator drift? How bounded is recursive self-improvement in current AI systems? How do standardized forms and measurement artifacts shape observable evidence? Do standard benchmarks reveal AI capability in real-world contexts? Does evolutionary optimization at test time improve model performance effectively? Can we reliably distinguish AI-generated from human-written text? How can we determine if computational systems are genuinely conscious?

Related concepts in this collection 1

This note in its neighbourhood — explore the map, then jump to a related concept in the list below.

Concept map
13 direct connections · 142 in 2-hop network ·dense cluster Open in graph ↗
Can models be smart without organized internal s… Can we track and steer personality shifts during m…

Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph

your link semantically near linked from elsewhere

Related papers in this collection 8

Papers most semantically related to this note, ranked by cosine similarity in the embedding space.

Search by related questions 4

Suggested questions this note speaks to — click to search the collection, or type your own.

Why do models fail under distribution shift if accuracy metrics stay high?Can steering vectors prove that representations are genuinely organized?Does scaling data automatically produce compositional reasoning or just better feature encoding?What test distinguishes genuine compositionality from fractured feature presence?
Original note title

identical performance metrics can mask fundamentally different internal representations — feature linear decodability does not guarantee representational organization