When developers let AI write code, how do they decide what to check carefully versus trust on faith?

This explores how developers actually decide between quick shortcuts (trusting tests, skimming code) and close, hands-on supervision when working with AI agents. The short answer from the corpus: developers almost never review everything. They rely on shortcuts by default. Detailed oversight tends to show up before the agent runs, not after. In interviews with 17 experienced developers, people treated passing tests as a stand-in for correct code, treated a sensible-looking plan as a stand-in for working code, and mostly skimmed the output Do developers actually review agent work exhaustively?. This isn't laziness. Checking every line an agent writes costs more attention than anyone has, so shortcuts are a rational response.

The surprising part is where the careful work goes. The same study found that oversight happens in four forms: setting up controls before the agent starts, planning together with it, watching it while it runs, and reviewing afterward. Developers put real effort into the early stages. They configure the agent and seed it with partial solutions so there's less to check later How do developers actually oversee AI coding agents?. A separate study of 19 developers found the same pattern and broke supervision into seven stages (Plan, Monitor, Wait, Review, Teach, Manual Fix, Update Assets). It also found that developers manage the load by planning carefully upfront, handing some oversight to other agents, and building reusable guidance files How do developers actually supervise AI coding agents?. So the choice isn't really heuristics versus rigor. Developers are rigorous early so they can use shortcuts late.

Other research suggests this instinct is correct. In a 48-person study of computer-use agents, plan-based oversight cut problematic actions by 76%. But once a risky action actually reached execution, users blocked only 21% of them, no matter which oversight strategy they used Does oversight strategy prevent risky agent actions or just catch them?. Watching in real time is where human attention is weakest. Tools that let people edit the plan step by step while the agent runs, assigning some steps to themselves and some to the agent, improved control without adding effort Can users steer AI agents better by revising plans mid-execution?.

There is also a third option besides 'skim everything' and 'check everything': let the system flag where it's unsure. In AutoResearchClaw, sending only the low-confidence decisions to a human got an 87.5% accept rate. Full autonomy got 25% and step-by-step oversight got only 50% Does targeted human oversight beat both full autonomy and exhaustive review?. Constant check-ins performed worse, apparently because people tired of them and started approving things without really looking. A related argument from AI safety research goes one step further. For actions that must never happen, the most reliable approach is to make them impossible in the system's design rather than counting on anyone to catch them Can architecture prevent violations better than training values?.

The takeaway you might not expect: the 'passing tests' shortcut is only as good as the effort that went into setting things up beforehand. Detailed oversight pays off most in planning and in the few decisions the system flags as uncertain, and least in watching an agent work in real time. Note that the corpus's direct evidence comes from small interview and lab studies (16–48 people), so treat these as strong patterns rather than settled rules.