Humans in the Loop: The Evolution of Work in Early Experiments with Generative AI

Source: Ben Armstrong, Julie Shah, with Kate Kellogg et al. (MIT Working Group on Generative AI and the Work of the Future); MIT Industrial Performance Center industry report (study of 20+ companies) · 2026

How can generative AI lead to better jobs? In January 2026, approximately half of American workers reported using AI, but how new technologies have affected the quality of their jobs remains largely unclear. Drawing on a study of more than twenty companies across four major industry groups – healthcare and life sciences; retail; finance, insurance, and real estate; and manufacturing – this paper identifies patterns in how organizations have experimented with generative AI; how those experiments have changed the roles of workers; and how organizations can support highquality jobs as they integrate new technologies.

The applications of generative AI among the companies we studied were directed toward three common challenges. The bottleneck problem is where workers are responsible for a growing number of simple tasks that get in the way of higher value-added work. Generative AI tools have been aimed at relieving these bottlenecks by speeding up near-routine tasks. The cafeteria problem emerges in a process that requires workers to consult experts from various domains and integrate their input into a product, document, or idea. Organizations have looked to generative AI to predict what those domain experts might have said based on what they have produced in the past. The learning curve problem represents the extra time and effort novices require to complete a complex task in a new domain. Generative AI tools have been directed to help workers to perform as if they had more experience – and to develop expertise – in new domains.

Across the applications of generative AI addressing these challenges, there has been a shift in the core tasks that professional and technical workers are being asked to perform. Where generative AI tools are being deployed, workers are increasingly asked to perform supervisory control tasks as the “human in the loop” overseeing and analyzing a process rather than executing the process manually. Although supervisory control tasks may be new to workers in law or healthcare, the “human in the loop” concept is not new.

Introduction. Soon after ChatGPT was released to the public in November 2022, Mark started peppering it with questions.

“What are you good at?”

“What makes you happy?”

“How do you have world peace with finite resources?

“Are you alive?”

Mark has a playful attitude toward new gadgets. An affable Midwesterner, he wasn’t afraid of ChatGPT as much as he was curious about how it actually worked – and what he could do with it.

As an engineering director at a Fortune 500 company, Mark was already familiar with AI. He had been working on integrating machine learning algorithms into his employer’s business for years. He had seen incremental progress, but nothing transformative.

Everyone was telling him that this time was different. Generative AI was going to change everything.

Part of him was skeptical of the hype. He had been burned before. The previous year, everyone in his field was saying that the metaverse was coming. He had worked on preparing his company to use VR headsets. That was a bust.

But at the same time, he was reading respected technology voices talking about Generative AI as a big step toward an Artificial General Intelligence where the algorithm could reason on its own, unprompted. He hadn’t heard people get this excited since the dawn of the internet. Mark wanted to see what was there -- how intelligent was this thing, really?

The answers the algorithm gave were nicely written and organized, but they weren’t that interesting. The way that ChatGPT described its skills was generic. Its response to his question about resource scarcity was word salad. It used human-like language, but it wasn’t anything like a human.

He kept poking and prodding, following up on the answers ChatGPT gave him. It became clear the bot wasn’t thinking on its own. He didn’t have to worry about anything like consciousness. But it could be an incredibly effective assistant.

He started asking the algorithm to rewrite emails and memos, to summarize complex documents for him. Then he started using it as he developed new software programs – to perform unit tests (a necessary task, but one he dreaded) and to help him develop a frontend for an app he was developing (an area that wasn’t his specialty). Each time he hit a bump or didn’t know the answer, he could ask the bot. It might not give him everything he needed, but it would get him moving again. Before, he might have gone down a rabbit hole, calling a friend or scrolling an online forum. Now, it was faster, and he felt he was capable of doing more.

What began as a curiosity became a legitimate hobby. Outside of work, Mark was watching YouTube videos and reading blogs about how best to prompt ChatGPT. He started teaching his team at work what it could do for them. It wasn’t just that the tool could help remove the drudgery – the repetitive coding – from their daily work. They could also develop new technology that reached customers.

Around the same time Mark and millions of others began experimenting with ChatGPT, the heads of companies large and small were imagining how the technology might change their businesses. They knew that rank-and-file workers, particularly engineers, were already playing with the technology and seeing how it could help them. They wanted to harness that creative spirit, but they also worried.

No one quite knew how the models worked, or why they gave the responses they did. You could prompt ChatGPT with the same question two times in a row, and it would give you different responses. Corporate leaders had heard that the technology could “hallucinate,” offering responses that were unreliable. They were also worried that if employees put confidential data into the model, it could show up in a response to a competitor’s query.

Many organizations put up walls, at least for a little while. Legal departments sent company-wide blasts with regulations asking companies not to use ChatGPT on the company network. The firewalls went up, and IPC Industry Report 4 companies formed task forces on how they could use the tool responsibly.

Three years after Mark began experimenting, organizations and individuals are still proving the concept of AI at work – laboring to transform processes and jobs beyond pilot demonstrations. It’s unclear for how long the ground is still going to be shifting beneath them. Bill Gates wrote in 1996, as Microsoft was taking off, “We always overestimate the change that will occur in the next two years and underestimate the change that will occur in the next ten.”

The generative AI transformations that companies have embarked on over the past three years are part of a decades-long journey. This paper is a chronicle of how that journey has begun – the experiments, false starts, and visions of those first years, as seen through the eyes of companies on the frontier of that journey.

Related work. 2. HYPOTHESES: HOW THIS TIME [MIGHT BE] DIFFERENT At a hearing under the stately arches of the Old Supreme Court chamber in the U.S. Capitol Building, lawmakers, business leaders, and scholars gathered for a special hearing on a technological breakthrough. Witness after witness described how new computing technologies promised to extend human capabilities and automate manual work. The upside was huge, they argued. Workers could produce far more per hour, and the technologies could help give rise to entirely new industries. One of the main questions was whether the United States could compete with its global foes and win the race to control the trajectory of the technology into the future.

While optimism filled the air, some urged caution. “If it were not for our present high-level employment, prosperous economic situation,” one Senator said, there might be “painful adjustments as a result of the great rush of technological change.” A statement from a labor leader emphasized that the change would require real adjustments from workers: what new skills would be required? What retraining would be offered? Would there be “large-scale social dislocations?”

“Questions of designing, installing and maintaining” are all problems that can be “solved by the engineer,” the labor leader’s statement read. “It is the human problems, the questions of adjustment to change, or preparation for new assignment...that cannot be solved in such a mechanical fashion but must be subject to the most imaginative thinking that all of us can give.”

The year was 1956. The labor leader was George Meany, then-President of the recently formed AFL-CIO.

Over the last 70 years, the public debate over new technologies and their impact on the economy have followed a familiar script. There is excitement about the prospects for economic growth and productivity on the one hand; concerns about what the change will mean for workers and incumbent industries on the other. An abiding question has been: how can new technologies support higher performance for employers, while also enabling higher-quality jobs for workers?

Wave after wave of technological change have produced consistent lessons. First, the impact of new technologies on jobs is often evolutionary – not revolutionary. Although technological breakthroughs can seem to introduce rapid change, the impact of those innovations often filters into businesses and the labor market slower as employers take time to transform their processes, redesign their jobs, and identify the most promising ways to use technology tools. Contrary to frequent predictions about massive job displacement due to automation technologies, technological change is often associated with incremental increases in some jobs and decreases in other jobs.

Method. 3. GENERATIVE AI MEETS REALITY In 2023, the MIT working group on generative AI and the work of the future began convening a group of companies to understand how they were approaching the adoption of generative AI tools. We originally targeted a small sample of companies to study their uses of generative AI, but soon we realized that far more companies were eager to share their experiences and learn from one another.

Between 2023 and 2025, we conducted dozens of interviews with AI leaders at more than 20 companies and convened more than six events with representatives from more than 50 companies to understand how the applications of generative AI tools were evolving. The companies that participated in the research and working group events were primarily large, established organizations that are among the leading firms in their industries – they were not primarily AI companies or startups aiming to unseat incumbent companies through their applications of AI. Many of the companies we studied were the incumbents themselves.

While we recognized that we were only studying the early stages of a long story, we sought to gather the perspective of businesses over time. How did AI and HR leaders within large companies discover what generative AI tools could do, and how did they learn from their experience? What did the rise of generative AI look like from the worker’s perspective – the people who began using these tools, and whose jobs were being affected?

At the outset we thought that the adoption and impact of these technologies would vary by sector, so we identified four categories of organizations that we thought would provide a cross-section of the economy: i) healthcare and life sciences, ii) retail, iii) finance, insurance, and real estate, and iv) manufacturing. We were interested in the different approaches to adoption in each of these domains, as well as the common threads that applied across industries.

We gathered data on the experiences of companies and their workers through several channels. We conducted interviews with senior leaders responsible for deciding how AI tools would be deployed, as well as the leaders charged with understanding how jobs and hiring would change due to generative AI deployments. Where possible, we spoke with individuals with direct experience using the applications of AI that had been deployed. Often the individuals we interviewed were managers within their organizations – not the frontline workers or individual contributors whose jobs have been affected by the implementation of AI tools.

We also draw on the results of a large-scale survey on workers’ attitudes toward and experience with AI and automation conducted in late 2023, as well as additional survey data of the public on AI usage patterns. These survey data serve to complement the interviews with management. Where the survey data aligns with what we learned from management, we can treat the interview data with additional credibility. Where the survey data and management interviews differ – for example, if managers suggest workers are enthusiastic about an application of AI, but survey data indicates worker opposition – then we can highlight the variation in perspectives.

In addition to interviews with participating companies, the research team convened a working group of more than 50 companies and other organizations to meet periodically to discuss progress and challenges that AI leaders faced as IPC Industry Report 11 they deployed generative AI tools. The meetings included five virtual sessions, as well as two in-person sessions, each of which had speakers from companies and other organizations who shared their work. Whereas interviews with organizational leaders and managers yielded detailed information about applications of AI, the working group sessions identified where organizational leaders saw consensus around the utility of these new tools, and where there were still unanswered questions. Throughout all these interactions with companies, we assured research subjects that they and their organization would remain anonymous unless they provide permission to publish their name.

Discussion.  ANALYSIS Our study of companies’ uses of generative AI provides initial evidence on the three hypotheses suggesting how this set of technologies might be different.

Whereas there continue to be predictions that generative AI adoption will have dramatic short-term displacement effects on certain segments of workers, we did not see evidence of this pattern in the companies we studied. Across the companies we studied, applications of generative AI were deployed primarily with a “human in the loop” and not associated with significant layoffs or changes to the structure of work within the firm or industry. In most cases, the adoption of these technologies, although faster than technologies of the past, still represented incremental change in the first three years (of course, this says nothing about what may happen in the next three years).

There were several domains where generative AI use has seen dramatic short-term adoption, including customer service, software engineering, and marketing and communication. In these fields, our interviews suggest, the adoption of generative AI technologies has changed the nature of jobs and the expectations for workers. In software engineering, generative AI tools have been associated with a slowdown in hiring of junior workers, although it is unclear whether the deceleration of hiring is due to actual effects of the technology – or anticipated effects.

It remains unclear whether even these domains where AI has seen the most impact are supportive of the shortterm displacement hypothesis. In customer service, for example, there is widespread use of generative AI tools, and only a small change in employment. Proponents of the hypothesis might point to declining job postings for customer service positions, similar to the relatively scarce job postings for entry-level software roles. However, the decline in customer service jobs (similar to the deceleration of software jobs) is oddly timed to the introduction of generative AI tools. For example, the decline of customer service job postings began a full year before ChatGPT was released.

Our observation that change to the workforce has been incremental is of course informed by the types of organizations that we studied – established firms, many in legacy industries, with large workforces that take time to transform. Even if these technologies were to have a dramatic long-term impact on the workforce in these industries, it would be unlikely that we would observe it in legacy companies in three years. It is possible that a transformative change will not happen by these companies creating new roles in some domains and shrinking roles in others – but by new companies emerging in these industries to compete with a completely different work organization.

Even with the incremental changes that we observed, there was a discernable change in the types of jobs that would be required if the organizations’ applications of generative AI were successful. These new jobs would ask workers to take on a greater share of supervisory control tasks, overseeing automated systems, and fewer tasks where they manually operate processes. These types of jobs already exist in automated environments. For example, pilots perform supervisory control tasks in cockpits; air traffic controllers and utility operators in control rooms; and technicians on factory floors all handle some combination of these tasks. However, these tasks are relatively new in fields like law and medicine.

For the skill leveling hypothesis, the evidence from early studies of generative AI is clear. suggests those with less skill and experience before performing a task with generative AI appear to experience higher performance gains when they use generative AI.

Conclusion. 4. LEVERS FOR SHAPING GENERATIVE AI There are three primary ways that applications of new technologies can fail to improve worker’s performance, or the quality of their work: disuse (not automating when adding new technology could be beneficial), misuse (deploying automation technologies with poor results), and overuse (the automation technologies work well, but they lead to unintended consequences). The applications of generative AI that we have studied are vulnerable to all three, but there are ways to guard against them.

Our findings from studying early real-world experiments with generative AI, coupled with historical data on technology and work, suggest ten practical lessons for employers, policymakers, and the workforce.

IPC Industry Report 27 The first lessons focus on the principles that can guide effective use of these tools.