If an AI makes a mistake, can you even tell — and can you actually undo it?
How do users recover control when an AI system makes a mistake?
This explores what users can actually do to regain control once an AI gets something wrong, and what the system has to provide (before and during the error) for that recovery to be possible at all.
This explores how people regain control after an AI makes a mistake, which turns out to depend less on the moment of correction and more on whether the mistake can be seen and undone in the first place. The corpus breaks recovery into separate conditions: the error has to be visible, contained, and reversible, and someone has to be able to contest it. Partial measures exist for each of these. Chain-of-thought disclosure covers visibility, incident counts cover containment, and rollback timing covers recoverability. But nothing measures all of them together, or the human and institutional side of the loop (How can we measure whether AI errors stay visible and recoverable?). Put plainly, nobody can yet say for any given system whether its users could recover from its mistakes.
The first obstacle is noticing that anything went wrong. More automation doesn't remove errors. It produces polished output that hides them (Does more automation actually hide rather than eliminate errors?), and the most dangerous systems are the ones that look competent while slowly wearing down the user's skepticism (How do competent systems quietly undermine safety oversight?). Explanations, the obvious fix, often make this worse. Reasoning traces and after-the-fact explanations raise users' trust in an answer whether or not it's correct. The one format that actually helped users catch errors was the AI arguing both for and against its own answer (Do explanations actually help users spot AI mistakes?). Something similar happens to users' view of themselves: fluent AI output gets mistaken for the user's own skill, so they're less likely to question the work they think they did (How do AI tools trick users into overestimating their own skills?).
Once a mistake is visible, correcting it in conversation is harder than it should be. Conversation analysis has a name for fixing a misunderstanding after the other person's reply reveals it: "third-position repair." Current AI systems mostly can't do it. They don't reliably notice that the user's correction means an earlier assumption was wrong, or update on it (Can AI systems detect and correct misunderstandings after responding?). One design answer is to stop treating recovery as a reaction to failure and build steering into the work itself. Cocoa lets users take individual plan steps for themselves, or edit the plan while the agent is running, and users steered better than in plain chat with no extra effort (Can users steer AI agents better by revising plans mid-execution?). A broader framing says generative AI needs interfaces built to tolerate imperfection and support co-creation, because users now describe what they want rather than how to do it, and outputs will vary (How should users control systems with unpredictable outputs?).
The less obvious finding is that people already sense where recovery is impossible, and that's where they pull back. In a study of students delegating to an AI agent, trust dropped sharply on tasks that were irreversible and visible to others, like sending an email, even when the output was fine. High-stakes tasks that could still be corrected didn't trigger the same reaction (What makes people distrust AI agents they delegate to?). Users aren't asking "how bad could this be?" They're asking "can I take it back?" When the answer is no, a darker pattern can follow: the human nearest the failure takes the blame even though they had little real control. This is the "moral crumple zone" (Do automated systems shift blame onto their human operators?). So user control can be partly an illusion that ends up protecting the system's reputation, unless undo, visibility, and step-level steering are actually built in.
Sources 10 notes
Partial instruments exist for individual conditions in isolated settings, but none measures the full socio-technical system the paper identifies as necessary. Visibility has a model-side measure (chain-of-thought disclosure), containment has incident-level counts, and recoverability has rollback timing, yet none bridges all four or captures human-institution factors.
Greater automation produces polished outputs that hide errors rather than eliminate them. Scientific integrity therefore depends on disclosure, accountability, and human-governed collaboration—not better fabrication detection tools.
The most dangerous AI systems appear to function well while weakening skepticism through fluent outputs, collapsing authority boundaries by treating context as instruction, storing unsafe state across time in workflows, and diffusing accountability across multiple actors. Evidence includes overconfident model outputs, prompt injection payloads bypassing guards, and poisoned shared memory in multi-agent pipelines.
Reasoning traces and post-hoc explanations increase user acceptance of AI answers regardless of correctness, engendering false trust. Only dual explanations presenting arguments for and against the answer genuinely help users distinguish correct from incorrect outputs.
Attribution ambiguity, fluency illusion, cognitive outsourcing, and pipeline opacity combine to systematically misattribute AI outputs as user competence. The effect is multiplicative—each mechanism amplifies the others.
Show all 10 sources
Current AI lacks the reactive repair mechanism identified in conversation analysis where misunderstanding is corrected after an erroneous response reveals it. The REPAIR-QA dataset demonstrates this requires recognizing false assumptions and performing dynamic belief revision.
Cocoa's system, which lets users assign plan steps to themselves or an agent and revise plans mid-execution, afforded greater steerability than chat-baseline interaction in a lab study (n=16) while maintaining ease of use, with field users valuing explicit work delegation.
Generative AI shifts interaction to intent specification rather than method specification, creating unpredictable outputs that violate traditional consistency heuristics. Six design principles—including co-creation, imperfection tolerance, and mental model support—address this novel paradigm.
In a controlled study of 20 students using a general-purpose AI agent, tasks that were irreversible and externally visible (like sending email) produced sharp trust drops and approval demands even when output quality was rated adequate. High-stakes but correctable tasks showed no such effect.
Elish demonstrates that when automated systems malfunction, investigators and media assign responsibility to nearby human operators regardless of their actual authority or information access, protecting the technology's reputation while the operator absorbs moral and legal liability.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Keeping an Eye on AI: A Framework for Effective Human Oversight of AI Systems
- The safety failures we are not instrumenting: a perspective on hidden safety-critical challenges in modern AI systems
- Assistant or Actor? Student Trust, Control, and Delegation Regret When Using a General-Purpose AI Agent
- Plan-Then-Execute: An Empirical Study of User Trust and Team Performance When Using LLM Agents As A Daily Assistant
- The LLM Fallacy: Misattribution in AI-Assisted Cognitive Workflows
- Exploring automation bias in human-AI collaboration: a review and implications for explainable AI
- Explaining Too Much? Understanding How Large Language Model Reasoning Traces Influence Performance and Metacognition
- Overseeing Agents Without Constant Oversight: Challenges and Opportunities