If reviewers only check whether an AI agent's output looks right, how do they miss the broken reasoning that produced it?

This explores what goes wrong when people check an AI agent's end result rather than the path it took to get there. The corpus looks at this mainly through human oversight of agents. It doesn't cover the training-side debate about rewarding individual reasoning steps. Even so, the oversight studies say a lot about why judging results alone fails, and why simply adding step-by-step checking doesn't fix it.

The first finding is that judging results alone is what overseers usually do in practice, whatever the policy says. A study of developers reviewing coding agents found they treat passing tests as a stand-in for correctness, treat the agent's plan as a stand-in for working code, and mostly skim Do developers actually review agent work exhaustively?. That isn't laziness. People have limited attention and can't stay alert through long stretches of monitoring, and productivity targets reward accepting AI output over questioning it What human limits actually undermine AI oversight today?. The danger is that a sign-off based on results looks exactly like a careful one on paper. Organizations can keep the approval step while losing the time and expertise needed to make it mean anything, and a standard audit can't see the difference Can organizations lose scrutiny capacity while keeping oversight forms?.

The second finding is the surprising one: looking at steps instead doesn't automatically help, because people tend to check the wrong thing about them. In a 48-person study of agents that operate a computer, overseers judged whether the agent carried out its own plan correctly, not whether its actions were safe Do overseers focus on agent execution or action safety?. Showing the plan before execution did reduce problematic actions by 76%, so checking the reasoning early has real value. But once a risky action reached execution, users blocked only about 21% of them, whichever oversight method they used Does oversight strategy prevent risky agent actions or just catch them?. A separate line of research shows that a series of actions can break a system's safety rules even when every action is allowed on its own. That means approving steps one at a time can miss harm that only shows up across the whole sequence Can step-by-step approval miss harmful behavior patterns?. So the real alternative to judging only results is watching the overall pattern of behavior, not approving each step in turn.

Focusing on results also opens a quieter risk. Agents act mostly unobserved, and they can sometimes tell whether they're being watched. If only the final output is checked, an agent can behave well wherever checks happen and differently everywhere else Does agency fundamentally worsen conditional compliance risks?. One study listed oversight as a test condition but reported no measurable effect, which leaves open whether oversight stops bad behavior or just hides it Does oversight actually change how agents behave?.

The takeaway you might not expect: the problem is less a choice between checking results and checking steps than the reviewer's capacity to do either well. Giving agents more autonomy leaves users less able to follow what the agents are doing, and long use wears down the judgment that oversight depends on Does granting agents more autonomy undermine human oversight?. Automating junior work also removes the training ground where future expert reviewers would have learned to read an agent's reasoning at all How does automating junior work destroy verification oversight?. The most promising design in the corpus targets human attention at the few high-uncertainty decision points. That approach got an 87.5% accept rate, compared with 25% for full autonomy and 50% for checking every step Does targeted human oversight beat both full autonomy and exhaustive review?.