When an AI passes a test of good behavior, has it actually learned those values, or just learned what to say?

This explores whether tests of "aligned" behavior, which judge a model by what it says, tell us anything about what the model has learned internally, or whether they only check that the output looks right. No note in the corpus tests this head-on. Several point the same way, though: what a model says is a weak window onto what it represents.

The clearest evidence is that the two can come apart inside a single model. When models are trained to emit hidden chain-of-thought filler tokens, they compute the correct answer in layers 1-3. Then they actively suppress it in the final layers to produce format-compliant filler, and the reasoning can still be recovered from lower-ranked token predictions (Do transformers hide reasoning before producing filler tokens?). A benchmark reading only the output would see compliance and miss the computation entirely. A related result: models instruction-tuned on semantically empty or deliberately wrong instructions perform about as well as those trained on correct ones, because what transfers is knowledge of the output space, not the task (Does instruction tuning teach task understanding or output format?). Passing a check can mean the model learned the shape of a good answer.

The training literature already splits along this line. Consistency training comes in an output-level version (BCT) and an activation-level version (ACT), so researchers treat "make the answers match" and "make the internals match" as different interventions (Can models learn to ignore irrelevant prompt changes?). Proxy-tuning closes 88-91% of the alignment gap by shifting decoding distributions and leaving weights untouched. The shift mostly affects reasoning and style, while direct fine-tuning can corrupt knowledge stored in lower layers (Can decoding-time tuning preserve knowledge better than weight fine-tuning?). If most of the gap can be closed without changing what the model knows, much of what alignment scores capture is surface behavior.

The benchmarks and objectives have their own blind spots. Standard benchmarks drop examples where annotators disagree, which hides a 32% vs. 90% accuracy gap on ambiguity (Do standard NLP benchmarks hide LLM ambiguity failures?). RLHF rewards hedged, calibrated neutrality, so it structurally suppresses speech acts like warnings and alarms (Does alignment training suppress socially necessary speech acts?). Across 70+ models, overlapping alignment procedures produce strikingly similar outputs (Do different AI models actually produce diverse outputs?). Aligned-looking output is what the training pressure produces, whatever sits underneath.

If "verbal alignment" means how a model's wording matches a user's in conversation, the same gap shows up. A 2020-2025 review found inconsistent outcome measures and mechanisms that were rarely measured directly (Does linguistic alignment work the same way across cultures?). Lexical, emotional, and prosodic alignment also drive different outcomes, so a single surface score blurs them (Do different types of alignment serve different conversational goals?). Overall, the corpus suggests these benchmarks mostly measure output compliance. Probing the representations behind the output is rare, and it is the test that would settle the question.