Does a faster trace interface actually improve error detection?

Across three user studies built around Magentic-UI, a human-in-the-loop computer-use agent (CUA) that can browse the web, run code, and use a file system, the paper tests how the design of an agent's "trace" — the text and screenshots that describe what the agent did — affects a person's ability to catch its errors. In the final study, a redesigned interface assembled from the two earlier studies' findings let participants find errors significantly faster than the existing trace format (Hedges' g: -0.65). But accuracy on the underlying task did not meaningfully improve (Hedges' g: 0.18), and — "more worrisome" in the paper's own words — when participants missed an error, they reported higher confidence in the agent's correctness (Hedges' g: 0.85) than participants working with the original trace. The paper states the result plainly: "participants found errors faster... However, we found no meaningful effect on improving overall accuracy... when errors were missed, participants reported higher confidence in the agent's correctness."

The mechanism traces back to the two earlier studies. A formative study (n=12) found the existing trace "verbose, cumbersome, and time-consuming," with participants missing "small but impactful errors" in long lists of individual actions. A second study (n=12) tested three summarizing probes — Flowchart, Citation, and Specification — presented after a task had already concluded. Flowchart, "the most process-oriented probe," had the lowest error-finding rate and was least preferred; Specification, an outcome-oriented summary, had the highest error-finding rate and was most preferred, but "took the longest" and "required the most checks of the overall trace." The final interface combined Specification's outcome framing with direct links into the execution trace, aiming to preserve a high-level overview while keeping it grounded in individual steps. Participants still justified wrong answers by citing a "reasonable" process the agent had followed, and the paper flags visible checkmarks for confirmed requirements as a plausible source of unearned confidence when a requirement was wrongly marked complete.

This sits as a caution against Can execution traces ground honest explanations of agent behavior?: grounding a summary in the execution trace is the design move this paper also makes, but doing so did not translate into better verification — only faster, more confident verification. It also bears on How should AI interfaces handle the shift from doing to supervising?: Magentic-UI's co-planning, co-tasking, and action-approval features are a working version of Nielsen's orchestration surface, and this paper's finding suggests that surface can satisfy users without actually improving their oversight.

The excerpt does not establish that these effects hold beyond this one system and population: participants were technology-company employees already experienced with AI agents, the interface was a Wizard-of-Oz design rather than a full production build, and the test measured only post-hoc verification of an already-completed task with a self-reported 1–7 confidence scale. Whether a better-navigated trace produces the same speed-without-accuracy, confidence-without-correctness pattern for other agent types (coding agents, deep-research tools) or for real-time rather than post-hoc oversight is left open by this study.