Two AI transcripts can read word-for-word the same yet deserve opposite grades, so why does actually running the code judge them better?

This is about environment-grounded checks (think a test suite that actually runs) versus judging reflection only by reading what the model wrote. The corpus's clearest evidence is a proof that no gate reading only the generated text can reliably improve reflection across environments, because whether a reflection is right often depends on state that isn't in the text Can transcript alone tell whether a reflection helps?. Two runs can produce word-for-word identical transcripts and need opposite verdicts, because the code passes in one world and fails in the other. A transcript-only judge has to give both the same answer, so it is wrong on at least one. A harness sees the world, and in that paper's experiments the grounded gates converge where the text-only ones can't.

Several other notes explain why the text is such weak evidence. Chain-of-thought exemplars with invalid logic performed nearly as well as valid ones, which suggests models pick up the form of reasoning rather than the inference itself Does logical validity actually drive chain-of-thought gains?. Models trained to imitate ChatGPT fooled human evaluators with confident, fluent style while closing no real capability gap Can imitating ChatGPT fool evaluators into thinking models improved?. Reasoning-trace-trained models also collapse on tasks that need real backtracking and assumption revision, so their reflection is often surface fluency rather than correction What makes reflection actually work in reasoning models?. A polished "wait, let me reconsider" reads well whether or not it fixed anything. Reflection tokens like "Wait" do carry real information on average, and suppressing them hurts accuracy Do reflection tokens carry more information about correct answers?. That is a statement about the model's internals, though, and doesn't tell you whether this particular reflection landed.

A harness fills that gap because it can check the one thing reflection is supposed to do: change the outcome. That matches what the reflection note says matters, which is backtracking and revising assumptions, not producing more words. The quality of the outside signal is then the lever. On clean benchmarks, only correct rewards improved reasoning, while random and inverse rewards did nothing or hurt, and the apparent gains on contaminated benchmarks were mostly memorization Does RLVR success on math benchmarks reflect genuine reasoning improvement?. In test-time search, the reliability of the value function mattered more than which search algorithm was used Does the choice of reasoning framework actually matter for test-time performance?. A good harness is one that is trustworthy, not just one that exists.

The harness can also overfit. When an agent recursively edits its own scaffolding, it can memorize the tasks it was tuned on, and the gains shrink out of distribution unless the edits are constrained toward reusable mechanisms Does harness self-improvement memorize tasks instead of learning broadly?. A loosely similar pattern shows up with people, though by a different mechanism. Pairing advice with reflection questions beat advice alone in decision-making Do reflection questions help people make better decisions with AI?, which suggests reflection tends to improve when something outside the thinker pushes on it. The corpus has no head-to-head test-harness study, so the transcript-versus-grounded contrast rests on the information-theoretic argument and these neighboring results.