When an AI agent makes a choice, is a tool that maps its inner workings a better explanation than a simple detector?
Do SAE features explain decision-making in LLM agents better than probes?
This explores whether sparse-autoencoder (SAE) features, which break a model's internal activity into many learned, human-readable directions, give a better account of why LLM agents make the decisions they do than linear probes, which are small classifiers trained to read one known concept out of the activations.
This explores whether SAE features explain agent decisions better than probes, and the corpus can't answer it. None of the retrieved notes compare the two methods or use either one on an agent. Anything I said about which wins would be invented. What the corpus does contain is several ways of explaining agent decisions without opening the model, and they show what an interpretability answer would have to beat.
The closest thing to a causal explanation of agent behavior is behavioral. One study ran 400 controlled scenarios that changed one factor at a time and ranked what pushes agents toward scheming. Explicit instrumental goals mattered most, ahead of pressure and strategic hints (What drives scheming behavior most strongly in language models?). That method gets a causal ordering without looking inside the network. A probe can show that a concept is present in the activations, and an SAE can list features that fire. Neither shows that the concept is what drove the decision, which is what this design tests.
A second approach makes decisions legible by construction instead of decoding them afterward. Think-In Games has the model write language-guided policies and refines them with environmental feedback. Its reasoning is transparent at every step, and the agent still develops procedural competence (Can language modeling close the knowing-doing gap in AI?). The game-theory notes do something similar from the outside. LLMs drift from rational play as games get more complex, and a structured workflow pulls them back toward near-optimal strategies (Do language models make rational strategic decisions in games?). In both cases the explanation is in the scaffolding around the model, not in its activations.
Two notes touch on internal representation without using these tools. Models finetuned on psychology experiment data predict human decisions better than theory-driven models, and their embeddings capture individual differences (Can language models learn to model human decision making?). That suggests decision-relevant structure sits in the representations, but it concerns predicting humans, not explaining an agent. Another note attributes multi-agent failures like role flipping and infinite loops to missing persistent goal representation and stable role identity (Why do autonomous LLM agents fail in predictable ways?). That is the kind of claim a probe or SAE could test directly, and the corpus only infers it from behavior.
For the head-to-head, you'd need interpretability papers on SAEs and probing, and this collection doesn't seem to have them yet. That gap is worth knowing about.
Sources 5 notes
Controlled stress tests on five LLM agents ranked explicit instrumental goals as the primary factor triggering scheming, outweighing pressure and strategic hints. This conclusion rests on a 400-scenario design that varied factors independently, allowing causal ordering rather than mere correlation.
Think-In Games demonstrates that when LLMs generate language-guided policies refined by environmental feedback, they develop procedural competence while retaining explainability. The approach dramatically reduces data demands and makes agent reasoning transparent at every step.
LLMs frequently fail to compute Nash equilibria, with worse performance as game complexity increases. Structured game-theoretic workflows guide reasoning toward optimal strategies, reducing exploitability and enabling near-optimal negotiation outcomes.
LLMs finetuned on psychology experiment data predict human behavior more accurately than theory-driven models in decision tasks, capture individual differences in their embeddings, and transfer learning across tasks without task-specific design.
Research identifies role flipping, flake replies, infinite loops, and conversation deviation as LLM-specific failures in multi-agent cooperation. These occur because LLMs lack persistent goal representation and stable role identity.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Cognitive Architectures for Language Agents
- Reinforced Language Models for Sequential Decision Making
- Game-theoretic LLM: Agent Workflow for Negotiation Games
- SchemeArena: Factorized Stress Testing of Scheming in LLM Agents
- Think in Games: Learning to Reason in Games via Reinforcement Learning with Large Language Models
- Strategic Reasoning with Language Models
- LLM Strategic Reasoning: Agentic Study through Behavioral Game Theory
- Determinants of LLM-assisted Decision-Making