Do overseers focus on agent execution or action safety?

The same 48-participant, 192-session study of computer-use agent (CUA) oversight strategies identifies a specific reason intervention stayed low even when control reached users at the right time: "users judged whether the agent acted correctly, not whether the action was safe" — a distinction the authors name "procedural correctness versus content appropriateness." Even when the moment to intervene arrived on schedule, people were checking that the agent had executed its own plan properly, not that the plan or action itself was safe to carry out.

The authors locate this failure inside a larger sequence: control is often returned too early or too late relative to when a risk actually arises, and the procedural/content split is what breaks even the cases where timing wasn't the problem. They connect it to a longer line of supervisory-control research — "preserving human authority does not, by itself, ensure effective intervention" — and to situation-awareness theory's distinction between perceiving information, comprehending it, and projecting its consequences. The authors argue overseers need what they call re-entry scaffolding, which reconstructs "the relevant antecedents, the consequential state transition, and the decision boundary," rather than just a description of the next action, if judgment is going to land on content instead of procedure.

This sharpens Does granting agents more autonomy undermine human oversight? by naming a specific cognitive failure mode rather than a general skills-erosion claim: the problem is not only that people lose the capacity to oversee, but that even attentive overseers default to the wrong evaluative question. It also qualifies the implicit premise of Does targeted human oversight beat both full autonomy and exhaustive review?, where routing a decision to a human assumes the human then judges it correctly — this study shows that assumption can fail even when routing timing is right, because the human may be answering "did this go as planned" rather than "should this happen at all." A related but distinct evaluative mismatch appears in What makes people distrust AI agents they delegate to?, where post-hoc regret tracked irreversibility and visibility rather than output quality; together the two studies suggest users apply the wrong criterion both during live oversight and in hindsight.

The excerpt does not report how often the procedural/content split occurred relative to timing failures, nor whether training, interface framing, or explicit safety checklists would shift users toward judging content instead of process — the authors offer re-entry scaffolding and "judgment alignment" as a design direction, not a tested fix. The implication is modest but concrete: an interface that invites the overseer to ask "was this action correct" is asking the wrong question, and oversight design may need to prompt "was this action safe" as a separate, explicit check.