A cheap lie-detector for AI cheating beat an AI watchdog on one model but lost on another. Why?

This explores why a cheap detector, a difference-of-means vector probe that flags reward hacking, can beat an LLM monitor on one model and lose to it on another. The corpus records the flip but doesn't explain it. On DeepSWE, the vectors caught 3.1% more hacks than LLM monitors in Kimi K3 but 7.9% fewer in GLM 5.2, at matched false positive rates How do cheap vector detectors compare to expensive LLM monitors?. That's two models on one benchmark, so nothing in the collection separates a real architectural effect from training differences or noise. What follows are the best-grounded hypotheses from neighboring notes, not established findings.

The strongest lead is that a vector probe assumes the model stores a concept, such as "I'm gaming this task," along one clean direction. One note argues that this assumption can hold without being reliable. Models can contain every linearly decodable feature a task needs while their internal organization is fractured, and the difference stays invisible to standard metrics Can models be smart without organized internal structure?. If two models organize the same concept with different tidiness, a probe that works well in one could be reading a smeared signal in the other. An LLM monitor reads the output text instead, so it wouldn't be affected. That would flip the comparison without the probe or the monitor changing.

The second lead is that "architecture" isn't one variable. Models differ in hidden size, the ratio of MLP to attention layers, and attention grouping, and those choices measurably change how they behave Can architecture choices improve inference efficiency without sacrificing accuracy?. Depth and width also change how concepts are built up across layers. Deep-and-thin designs compose abstract ideas through many layers, which suggests that where a concept becomes readable depends on the network's shape Does depth matter more than width for tiny language models?. A probe placed at a fixed layer or fitted the same way could land in a good spot in one model and a poor one in another.

Training may matter as much as architecture. Reasoning and non-reasoning models stay apart even with unlimited inference compute, because training installs a different way of working, not just different capacity Can non-reasoning models catch up with more compute?. Even a plain benchmark can produce qualitatively different curves by model type: small models degrade linearly, mid-range ones exponentially, and reasoning models hold steady and then fall off a cliff How does instruction density affect model performance?. A method's ranking against a baseline is a curve too, so it may not carry over from one model family to another. A related result shows the same pattern in a different setting. Removing spurious cues hurts performance on some tasks, the opposite of what a filtering story predicts, because the model's failure lies in how it combines signals Why does removing spurious cues sometimes hurt model performance?.

The practical takeaway is that a detector's benchmark result belongs to a specific model, not to the detector alone. If you're choosing between a free probe and a paid monitor, test the choice on the model you'll deploy it on. The corpus doesn't yet have the experiment that would isolate why the sign flips, such as the same probe across controlled architecture variants.