If you judge an AI by what its actions do rather than the reasoning it writes out, what do you lose?
What does effect-based monitoring sacrifice compared to language-based CoT monitoring?
This explores what you give up by judging an AI system from what its actions actually do (effect-based monitoring) instead of from the reasoning it writes out (language-based chain-of-thought monitoring).
This explores what you give up by judging an AI system from what its actions actually do, instead of from the reasoning it writes out. The corpus has no note that compares the two head-to-head, so what follows is partly inference from neighbouring material. The closest note, Does added monitoring improve protection at acceptable cost?, describes a comparison of monitoring designs (isolated actions, rolling windows, known groups, discovered episodes) at equal review cost. It reports no results, so it can't say what any of them loses.
The likely sacrifice is a window into intent. Reading a chain of thought lets a monitor see what the model says it is planning before anything happens. It also shows why the model chose an action, which makes a suspicious pattern easier to interpret. Effect-based monitoring only sees what has already been done. It can flag that something went wrong, but it can't tell a mistake from a plan, and it can't act before the effect exists. That reading is my inference. The corpus doesn't test it.
The corpus also suggests language-based monitoring is a shakier trade than it looks. Several notes argue that written reasoning is a weak view into what the model is actually doing. Does chain-of-thought reasoning reveal genuine inference or pattern matching? says chain of thought reproduces familiar reasoning patterns from training rather than reflecting genuine inference. Does logical validity actually drive chain-of-thought gains? found that illogical reasoning examples gain almost as much as valid ones. That means the text's logic isn't what drives the answer. What three separate factors drive chain-of-thought performance? splits chain-of-thought performance into output probability, memorization and noisy step-by-step reasoning, so a trace mixes real reasoning with other influences that a reader can't separate.
The sharpest hint is in Why does chain-of-thought reasoning fail in predictable ways?, which notes that performance optimizes against interpretability. If training rewards traces that look coherent, the text can stay readable while becoming a worse guide to behavior. Effect-based monitoring gives up the readable explanation but doesn't depend on the model's account of itself being honest or faithful. Whether that beats reading the reasoning is exactly what the corpus doesn't test.
Sources 5 notes
The paper designs a controlled comparison of isolated actions, rolling windows, known groups, and prospectively discovered episodes at equal review cost and false-alert workload, but the excerpt provides no empirical results showing whether added monitoring improves protection.
CoT works by constraining models to reproduce familiar reasoning patterns from training, not by enabling novel symbolic reasoning. Performance degrades predictably under distribution shifts—the signature of imitation rather than capability emergence.
Illogical chain-of-thought exemplars matched valid CoT performance on BIG-Bench Hard, showing that structural properties—not logical validity—drive the gains. The model learns the form of reasoning, not genuine inference.
A shift cipher study decomposed CoT into three independent factors: output probability alone swings accuracy from 26% to 70%, memorization matches pre-training frequency patterns, and genuine reasoning exists but accumulates error with each step. This resolves the reason-or-memorize debate by showing LLMs do both simultaneously.
CoT guides models to pattern-match reasoning structure rather than perform genuine inference. This explains distribution-bounded failures, why structural coherence matters more than content correctness, and why performance optimizes against interpretability.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Is Chain-of-Thought Reasoning of LLMs a Mirage? A Data Distribution Lens
- When More is Less: Understanding Chain-of-Thought Length in LLMs
- CoT is Not True Reasoning, It Is Just a Tight Constraint to Imitate: A Theory Perspective
- Break the Chain: Large Language Models Can be Shortcut Reasoners
- Deciphering the Factors Influencing the Efficacy of Chain-of-Thought: Probability, Memorization, and Noisy Reasoning
- Hierarchical Reasoning Model
- Ring-Zero: Scaling Zero RL to a Trillion Parameters for Emergent Reasoning
- Measuring Faithfulness in Chain-of-Thought Reasoning