Agentic AI and Human-in-the-Loop Interventions: Field Experimental Evidence from Alibaba's Customer Service Operations

Abstract Agentic AI systems that autonomously perform service tasks are entering customer service operations.

However, limited evidence exists on how human interventions shape service outcomes when agentic AI failures create both cognitive and emotional consequences. We study this issue through a randomized field experiment on Alibaba’s Taobao platform. Workers in the treatment condition supervised an agentic AI system that resolved AI-eligible chats while continuing to handle AI-ineligible chats, whereas control workers resolved all chats without agentic AI. The findings show that AI deployment reduces average chat duration and has limited effects on retrial rates, but substantially lowers ratings for AI- eligible chats. Moreover, human intervention effectiveness in AI-eligible chats depends on the nature of AI failure, post-escalation intervention effort, and intervention timing. Human intervention preserves service quality in algorithm-triggered technical escalations, i.e., unresolved customer issues beyond the AI’s capability, but is less effective in algorithm-triggered emotional escalations, i.e., where customers express frustration or dissatisfaction. These differences are partly explained by variation in workers’ post-escalation intervention effort across escalation types. In algorithm-triggered emotional escalations, workers showed lower engagement: they sent fewer messages, contributed a smaller share of total chat rounds, and showed less proactivity in information seeking and solution provision. We further find that early intervention is essential for sustaining high post-escalation intervention effort. Finally, we document a positive spillover effect on AI-ineligible chats, as treated workers adapted their multitasking workflow to devote greater attention to these chats. These findings offer implications for human-in-theloop process design in human-AI collaboration systems.

Introduction. Recent advances in generative AI have given rise to a new class of agentic AI systems that can not only generate content but also execute tasks traditionally performed by human workers. In a recent survey of senior executives at leading companies, 35% of respondents reported having adopted AI agents by 2023, and another 44% indicated plans to deploy them in the near term (Ransbotham et al. 2025). This development has attracted substantial attention in customer service. Unlike earlier automation technologies, which supported workers at discrete stages of the service process, agentic AI can engage in turn-by-turn communication to interpret customer inquiries, determine appropriate next actions, and carry out service tasks in real time. However, the technology remains immature in important respects. In particular, agentic AI still faces limitations in adaptability and contextual judgment, which prevent it from reliably resolving all customer service requests.

Human-in-the-loop intervention therefore remains an important safeguard. In this study, we examine how such intervention shapes service outcomes.

Our study departs in important ways from the existing literature on human-in-the-loop systems, which has largely focused on engineering settings such as medical imaging and warehouse operations. In these contexts, AI failures are predominantly cognitive in nature, and human intervention typically involves correcting well-defined errors (Sheridan and Parasuraman 2005, McKinney et al. 2020, Sun et al. 2022). By contrast, customer service represents a fundamentally different operational environment. The task is not only cognitive but also requires workers to manage customer emotions and sustain interpersonal rapport throughout the interaction. As a result, service failures in this setting may induce both cognitive and emotional consequences.

This distinction raises three unresolved questions: (1) How does deploying agentic AI under human-in-the-loop intervention affect customer service performance? (2) When and how can human interventions be effective? (3) How should firms design the process of human-agentic AI systems? Addressing these questions is critical for understanding the performance implications of deploying agentic AI in customer service operations.

We examine these questions using a randomized field experiment conducted on Alibaba’s ecommerce platform, Taobao.com, in August 2024, involving 647 customer service workers and 680,676 service chats. At the time of the experiment, Alibaba had deployed an agentic AI system capable of autonomously resolving a subset of standardized service chats that were classified as AI-eligible. Otherwise, chats were classified as AI-ineligible, which required full human resolution.

Workers assigned to the treatment condition supervised AI-eligible chats and intervened when necessary, while continuing to resolve AI-ineligible chats themselves. Workers in the control condition fully handled all chats throughout the study period.

Using worker-level difference-in-differences analysis, we find that the deployment of agentic AI reduces the average duration of chats and has limited effects on retrial rates, but substantially decreases customer ratings for AI-eligible chats. Furthermore, the results demonstrate that the effectiveness of human intervention on AI-eligible chats depends on the nature of AI failure, the post-escalation human intervention effort, and the timing of intervention.

Human intervention preserves service quality in algorithm-triggered technical escalations, i.e., unresolved customer issues beyond the AI’s capability, but is less effective in algorithm-triggered emotional escalations, i.e., cases where customers have expressed frustration or dissatisfaction.

These differences are partially explained by variations in the post-escalation intervention effort of human workers across different escalation types. For emotional escalations triggered by algorithms, workers exhibited lower engagement—they sent fewer messages, contributed to a smaller share of total chat rounds, and demonstrated less proactivity in information seeking and solution provision.

The timing of human intervention plays a central role in explaining when human-in-the-loop systems succeed or fail. In technical escalations, human workers intervene while the customer issue remains primarily a capability mismatch, leaving sufficient room for recovery. In emotional escalations, by contrast, intervention often occurs only after customer frustration has accumulated, making the interaction harder to repair and reducing workers’ subsequent engagement. Humaninitiated escalations suggest that earlier intervention can partially prevent this deterioration: when supervisors take over before negative sentiment becomes entrenched, they are better able to sustain effort and limit the decline in service quality.

Related work. 2.1 Agentic AI in Service Operations Recent advances in AI have shifted its role in service operations from an assistive tool to an autonomous agent. Prior research on AI in operations has primarily focused on prediction, reasoning, and content-generation tools that support human workers, including generative AI (Noy and Zhang 2023, Chen and Chan 2024, Brynjolfsson et al. 2025, Ni et al. 2026, Bai et al. 2026). In these settings, AI augments human labor by generating suggestions, drafting content, or synthesizing information, while humans remain the primary decision makers and communicators.

Agentic AI differs in an important way: rather than merely supporting human work, it can autonomously execute multistep tasks and directly interact with customers in real time. The central question is therefore no longer whether AI assistance improves human performance, but whether service systems perform effectively when agentic AI handles frontline labor and humans act as AI supervisors. Recent modeling work suggests that such systems may also reshape organizational structure by altering task allocation between different workers (Xu et al. 2025). Empirical evidence on these issues, however, remains scarce, particularly in customer-facing settings. This study contributes to addressing this gap by providing field experimental evidence from a real service environment in which agentic AI interacts directly with customers under human supervision. Our findings have important implications for how firms redesign the organizational structure and human labor in deploying agentic AI systems.

2.2 Human-in-the-Loop Interventions in Human-AI Service Systems Secondly, we contribute to the literature on human-in-the-loop systems, which emphasizes the role of human intervention in improving system reliability by correcting AI errors and preserving accountability (Sheridan and Parasuraman 2005, McKinney et al. 2020, Sun et al. 2022, Lebovitz et al. 2022, Benjaafar et al. 2025). Most prior work, however, focuses on engineering settings such as autopilot systems, robotics, and medical AI, where failures are primarily cognitive and interventions are designed to correct well-defined errors.

Customer service differs in an important respect because failures of AI can have both cognitive and emotional consequences. When agentic AI interacts directly with customers, failures may accumulate through miscommunication and rising customer frustration before human intervention occurs. Consequently, the effectiveness of human intervention may depend not only on whether AI fails, but also on the type of failure and the timing of intervention.

Method. In May 2024, Alibaba developed an agentic AI system to support online customer service. The system uses a large language model (LLM) as its core decision engine to coordinate multiple generative AI functionalities and autonomously manage service chats. The deployment also introduced a redesigned process for handling online service chats, as illustrated in Figure 1.

Figure 1: Customer Service Process with the Agentic AI System Following the deployment of agentic AI, incoming chats were classified as either AI-eligible or AI-ineligible. AI-eligible chats involved standardized customer issues that could be resolved autonomously by the agentic AI system, whereas AI-ineligible chats involved non-standardized issues that required full human resolution.

At the time of the experiment, AI-eligible chats accounted for less than 10 percent of total chat volume. Even within this subset, the agentic AI system could not reliably complete all chats because of limitations in adaptability and interpersonal communication. To address these limitations, Alibaba embedded the agentic AI system within a human-in-the-loop structure. Each AI-eligible chat was assigned to a designated human worker who supervised the interaction and retained responsibility for service completion.

The supervisory role of human workers involved two core functions: monitoring and intervention. Supervisors could monitor AI-eligible chats through a dedicated interface. Because human monitoring capacity was limited, Alibaba also implemented algorithmic monitoring tools that continuously tracked the workflow of the AI system and flagged chats with an elevated risk of service failure. When such risk was detected by the algorithm, the chats were escalated to human supervisors. Alternatively, human supervisors may trigger escalations by themselves based on their own judgment. Once escalations are triggered, human supervisors then intervene in the workflow of the AI system, take over the chats, and complete the chats.

3.4 Experiment Design Alibaba conducted a randomized field experiment from August 15 to August 31, 2024, to evaluate the impact of deploying the agentic AI system. The experiment involved 647 randomly selected customer service workers and generated 680,676 online service chats during the study period.

Workers were randomly assigned to treatment based on their employee ID, resulting in 345 workers (53.3%) in the control group and 302 workers (46.7%) in the treatment group. Random assignment at the worker level ensures baseline comparability across groups, which we verify through balance checks as reported in Section 3.5.

Figure 2: Experiment Timeline Figure 2 presents the timeline of the experiment. The first 14 days, from August 1 to August 14, 2024, constitute the pretreatment period, during which the agentic AI system was not yet deployed and all service chats were handled by human workers. The following 17 days, from August 15 to August 31, 2024, constitute the treatment period, during which the agentic AI system was deployed exclusively for workers in the treatment group. During this period, treated workers supervised AI-eligible chats while continuing to handle AI-ineligible chats themselves. By contrast, control workers fully handled all chats. The experiment ended on August 31, 2024, after which the agentic AI system was rolled out to all workers beginning the next day.

Compensation under the redesigned process also varied by chat type. For AI-ineligible chats, workers continued to receive piece-rate compensation based on chat volume, multiplied by a quality-based bonus coefficient. For AI-eligible chats, workers received compensation based on chat volume regardless of whether the chat was handled by the agentic AI system under human supervision or by human workers alone. Regardless of who eventually resolved the customer issue, standardized operating procedures (SOPs) are used. AI-eligible and AI-ineligible chats also contributed equally to workload assessments.

Conclusion. and Discussion This study examines a central question in the deployment of agentic AI in customer service operations: when autonomous AI systems interact directly with customers, under what conditions can human-in-the-loop intervention effectively recover service failures? Using a randomized field experiment on Alibaba’s Taobao platform, we show that the performance consequences of agentic AI deployment are fundamentally heterogeneous. On the one hand, agentic AI improves service speed and generates substantial efficiency gains, especially in AI-eligible chats. On the other hand, these gains do not uniformly translate into better service outcomes. While the objective service quality measure, retrial rates, remains largely unchanged, the subjective service quality measure, customer ratings, diverges sharply across chat types, declining in AI-eligible chats but improving in AI-ineligible chats.

These findings contribute to a more nuanced understanding of human-AI collaboration in service operations. Existing research on human-in-the-loop systems has largely focused on technical settings in which AI failures are primarily cognitive and human intervention serves as a safeguard against well-defined errors. Our results show that customer service poses a different challenge. In this setting, failures may involve not only technical mismatch, but also emotional deterioration that unfolds during the chat. Human intervention is effective when the failure is primarily technical, and it is also more effective when workers intervene proactively before customer sentiment has fully deteriorated. By contrast, once the interaction has accumulated substantial negative emotions, the scope for recovery narrows considerably.

The evidence further suggests that this weaker performance of human-in-the-loop intervention in emotional escalations is linked to differences in post-escalation human effort. When workers take over emotionally escalated chats, they contribute fewer messages, account for a smaller share of total chat rounds, and show less initiative in information seeking and solution provision.