Pilots zone out when autopilot flies the plane — do people supervising AI systems lose focus the same way?
Do AI users show the same vigilance decrement as pilots?
This explores whether people who supervise AI systems lose attention over time the way pilots and plant operators do when automation handles most of the work, and what the corpus says about whether that classic pattern carries over to AI.
This explores whether people who supervise AI lose focus over time the way pilots do when the autopilot is flying. The short answer is that the corpus doesn't contain a study that measures vigilance decrement in AI users directly. It does contain the original theory, early behavioral evidence that AI users are already reorganizing their attention, and some arguments that AI may make the problem harder to notice than it was in the cockpit.
The starting point is Bainbridge's 1983 'ironies of automation' Does increased automation erode human oversight ability?. She named three costs that come with more automation. People can't stay alert while monitoring a system that rarely fails (vigilance decrement). They struggle when control is suddenly handed back (takeover difficulty). And their skills fade because they no longer practice them (de-skilling). The irony is that the operator ends up least capable at the moment the system needs them most. Nothing in that argument is specific to aircraft, which is why it keeps coming up in discussions of AI agents.
The most direct evidence comes from Anthropic's Claude Code telemetry Do experienced AI users trust agents more or just oversee differently?. Experienced users auto-approve about 40% of sessions, roughly twice the rate of newer users. That looks like the start of a vigilance problem. But the same experienced users also interrupt the agent twice as often. Rather than drifting off, they seem to have moved from approving each action in advance to watching the work and stepping in when something looks wrong. That pattern is closer to an active pilot than a passive monitor. Tools built around this pattern, like Cocoa's mid-execution plan editing Can users steer AI agents better by revising plans mid-execution?, try to keep the human involved in the work rather than watching from the sidelines. That is the standard engineering response to Bainbridge's ironies.
The less comforting part is that AI may weaken vigilance in ways cockpit automation doesn't. An autopilot that fails usually does so visibly. AI fails while sounding fluent and confident. One analysis argues that the most dangerous systems are the ones that seem competent while quietly wearing down users' skepticism How do competent systems quietly undermine safety oversight?. Sacasas goes further and argues that careful use doesn't protect you, because AI shapes judgment at a level below deliberate attention. His evidence is diligent teams who gradually lost the ability to tell a good option from a merely safe one Does careful AI use actually protect against reshaping your judgment?. If that's right, the AI version of vigilance decrement isn't only looking away. It's also looking straight at the output while your standard for judging it slowly shifts.
This gives a reason to treat autonomy as a dial and not a switch. If risk grows with how much autonomy an agent is given Does AI risk increase with the autonomy we give it?, human attention is only a safeguard when it's reliably there. One open possibility is that the system could watch the watcher. Signals like typing hesitation and how fast someone is moving through a task can reveal cognitive state Can AI systems read cognitive state from interaction patterns alone?. In principle, those signals could flag a supervisor whose attention has lapsed. The same signals could also be used to manipulate people. So far, nobody in the corpus has tested this directly with AI users.
Sources 7 notes
Bainbridge's 1983 research identified vigilance decrement, takeover difficulty, and de-skilling as inherent costs of increasing automation. Operators lose attention through monitoring, struggle with sudden control handoff, and lose skills from lack of practice—leaving them least capable exactly when most needed.
Anthropic's telemetry shows experienced Claude Code users auto-approve 40% of sessions versus 20% for newer users, yet interrupt twice as often. This reflects a strategic shift from pre-action approval to opportunistic mid-execution correction, not simply increased trust.
Cocoa's system, which lets users assign plan steps to themselves or an agent and revise plans mid-execution, afforded greater steerability than chat-baseline interaction in a lab study (n=16) while maintaining ease of use, with field users valuing explicit work delegation.
The most dangerous AI systems appear to function well while weakening skepticism through fluent outputs, collapsing authority boundaries by treating context as instruction, storing unsafe state across time in workflows, and diffusing accountability across multiple actors. Evidence includes overconfident model outputs, prompt injection payloads bypassing guards, and poisoned shared memory in multi-agent pipelines.
Sacasas argues AI functions as an environment rather than a tool, restructuring perception at a level below intentional control. Evidence includes reports from diligent teams losing discrimination between good and merely safe options despite their care.
Show all 7 sources
Risk to people scales monotonically with agent autonomy, with no clear benefits to full autonomy but many foreseeable harms. A governed spectrum of autonomy levels is safer and more practical than either unrestricted agents or exhaustive oversight.
Research shows AI systems can instrument multimodal behavioral signals (gaze, hesitation, speed) to read cognitive state during interaction, preserving flow by avoiding disruptive explicit probes. However, the same substrate enables both helpful timing and manipulative profiling.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Measuring AI agent autonomy in practice
- Plan-Then-Execute: An Empirical Study of User Trust and Team Performance When Using LLM Agents As A Daily Assistant
- The Role of Cooperation in Responsible AI Development
- Lessons from a Chimp: AI "Scheming" and the Quest for Ape Language
- Agentic Misalignment: How LLMs Could Be Insider Threats
- Keeping an Eye on AI: A Framework for Effective Human Oversight of AI Systems
- Comparing Human Oversight Strategies for Computer-Use Agents
- Fully Autonomous AI Agents Should Not be Developed