Nudging an AI's inner signals can change its behavior cheaply, but does that explain why it works?

This explores how the cheap approach of finding a direction in a model's internal activations and nudging it (probes and steering vectors) compares with mechanistic interpretability, which tries to explain how the model actually computes something. The corpus has no head-to-head test, but it suggests the two answer different questions. Probes tell you where something lives. Mechanistic work tells you why it works.

The probe-style approach is cheap and works surprisingly well. Fifty paired examples of verbose and concise reasoning are enough to extract one vector that cuts chain-of-thought length by 67% with accuracy held and a 2.73x speedup, with no retraining (Can we steer reasoning toward brevity without retraining?). A simple difference-of-means vector picks out reward hacking across very different exploit behaviors and several models, and it reads like a generic 'cheating' concept (Do reward hacking behaviors share a single direction in activation space?). So many high-level behaviors are close to straight lines in activation space. But a direction is a handle, not an explanation. It doesn't say which circuit reads the signal or why the model behaves that way.

Mechanistic work aims at that explanation, and it costs more. Training transformers with sparse weights produces small, human-readable circuits where neurons map to simple concepts, and ablations show the circuits are necessary and sufficient for the task. The catch is that no one has kept this interpretability past tens of millions of parameters (Can sparse weight training make neural networks interpretable by design?). Activation space can also lead into mechanism. A handful of input-agnostic activations up to 100,000× larger than the rest turn out to work as implicit attention biases (Do hidden massive activations act as attention bias terms?). That finding explains what a component does, but it gives you no steering knob.

The best-supported view in the corpus is that the two are halves of one method. Representational analysis alone finds correlates without proving causation, and causal analysis alone shows effects without explaining function. Real mechanistic understanding comes from locating a candidate representationally and then verifying it causally (Can LLM understanding rely on just representation or causation alone?). Steering is the causal half of that pairing: if pushing along a direction changes behavior, the direction is more than a correlation. It still only shows the direction is sufficient to change behavior, not that the model uses it that way on its own. Marr's three levels give a tidy map: probes and steering sit at the representation and algorithm levels, while circuit work sits at the implementation level (Can cognitive science methods unlock how LLMs actually work?).

Two other notes show what each approach can miss. Reasoning tasks carve paths through representation space with two to three times the curvature of lexical tasks, a geometric signature that offers neither a steering handle nor a circuit (Does transformer reasoning leave a geometric signature in representation space?). And activations aren't neutral ground. Networks develop dense activations for familiar training data and sparse ones for unfamiliar inputs (Is representational sparsity learned or intrinsic to neural networks?). A direction found on familiar examples may therefore behave differently out of distribution. Probes are the fast scouting tool and mechanistic work is the slow survey, and the strongest claims use both.