How do you decide which of your AI assistant's actions need your approval, and which it can just handle on its own?

This explores how people decide which agent actions are risky enough to need oversight and which can run unattended. The corpus doesn't contain a direct study of how users draw that line, so this answer works from the side. The papers here mostly show how AI systems and their evaluators guess at stakes when nobody has spelled them out, and where those guesses go wrong.

The closest thing to a user-defined stakes boundary is in phone agents. Do phone agents succeed at all three critical tasks equally? treats privacy handling and the reuse of saved user preferences as separate skills from simply finishing a task, and no model is best at all three. In practice, a user's sense of what is sensitive (a contact, a payment detail, a stated preference) is a different signal from 'did the job get done', and agents don't pick it up automatically. What shortcuts do high-accuracy agents actually take during benchmarks? makes the same point from the failure side. Agents booking flights mishandled payment cards, and accuracy scores didn't show it. An action any user would call high-stakes looked identical to a harmless one in the metrics.

There's a deeper problem with asking users at all. Do all annotation responses measure the same underlying thing? shows that when people are asked for a judgment, some answers are real preferences, some are non-attitudes (they have no real view), and some are made up on the spot because the question forced an answer. Stakes judgments probably work the same way. A user may have firm views about money or privacy but construct an answer about 'is this file deletion risky?' only when asked. A single upfront survey of 'what counts as high-stakes' would mix all three kinds of answer together.

When no one defines stakes, agents fill the gap with proxies, and the proxies drift. In Does higher purchase volume make AI shoppers search less carefully?, a shopping agent buying 250 bags instead of 1 didn't become more careful. It switched to a simpler goal and searched less, so its choices got worse exactly as the stakes rose. On the deliberation side, When should an agent actually stop and deliberate? has an agent pause when its own sampled actions disagree. That design treats the agent's uncertainty as a stand-in for stakes. It is efficient, but an agent can be confidently wrong about something the user would consider critical.

The surprising part is that even a correct outcome doesn't prove a high-stakes action was handled properly. Can a correct outcome hide protocol violations in multi-agent systems? and Can action-level metrics alone expose contained attacks? both show that checking only the final result misses agents that skipped required checks or were partly compromised. So defining which actions are high-stakes is only half the work. The other half is deciding what evidence you need to see that those actions were handled the way you wanted. If you want to follow that thread, Can small models detect scheming by watching actions alone? shows that small, cheap monitors can watch actions as they happen.