Can AI tools meant to assist workers quietly turn into autopilot once nobody has the skills left to check them?

This asks whether AI meant to help workers can slowly become AI that works with no one really in charge, as the people 'in the loop' lose the skills to catch its mistakes. The corpus says yes, and that the slide happens quietly. A study that mapped 8,356 workplace AI risk scenarios found that augmentation offers no built-in protection. When workers lean heavily on AI agents, their skills and their ability to oversee the work wear down together. That removes the main reason augmentation was considered the safe option in the first place Does AI augmentation protect workers from skill erosion?. Nobody decides to automate. The human is still technically there, but they no longer do anything meaningful.

The way this erosion works was documented long before LLMs. A meta-analysis of 18 automation experiments found that automation improves everyday performance but makes people worse at recovering when it fails. The damage is sharpest at one point: when the system moves from analyzing information to choosing actions When does automation harm performance after failures?. People who watch a system make decisions lose the awareness and practice they'd need to step in. That matters for today's AI agents, which are moving from 'suggest' to 'do.' A field experiment on Alibaba's Taobao adds a timing lesson. Human takeovers fixed AI failures when they came early and the problem was technical. They failed when they came late, after customer frustration had built up When human workers take over from AI, does it actually fix the problem?. Oversight has to be skilled and also arrive at the right moment, and both depend on practice.

The idea that augmentation and automation are separate choices also turns out to be misleading. Raisch and Krakowski argue that they look like opposites at the level of a single task. Across many tasks and over time, they feed each other, with each creating the conditions for the other Are automation and augmentation really a simple trade-off?. Augmentation generates the data, routines and trust that make automation possible later. The drift your question describes is less a malfunction than a built-in tendency.

Economic incentives push in the same direction. Acemoglu, Autor and Johnson argue that firms earn more by automating expertise than by building tools that create new work for people, so pro-worker AI gets too little investment Why do firms build automating AI instead of pro-worker AI?. Brynjolfsson warns that automation moves bargaining power to whoever controls the technology, which can trap workers without leverage Does automating labor concentrate power more than augmenting it?. Scaled up to society, the 'gradual disempowerment' argument holds that institutions stay aligned with human interests partly because they depend on people who care about the outcomes. As AI takes over that work, the hidden check weakens, possibly beyond the point of reversal Does incremental AI replacement erode human influence over society?. The same erosion that happens to one worker's skills can happen to society's ability to steer.

The drift isn't guaranteed, though. Gans and Goldfarb show that when tasks depend on each other, automating some of them focuses human effort on the bottlenecks that remain and makes that work more valuable Does automating some tasks actually increase the value of remaining work?. Firm-level data shows that when AI affects only a few tasks in a job, workers can move to the tasks it doesn't touch Does concentrated AI exposure enable workers to adapt and reallocate?. The useful question is less 'augment or automate?' and more 'which human skills stay in daily practice?' Oversight that people never exercise is the first thing to disappear.