Complementarity in Human-AI Collaboration: Concept, Sources, and Evidence

ABSTRACT Artificial intelligence (AI) has the potential to significantly enhance human performance across various domains. Ideally, collaboration between humans and AI should result in complementary team performance (CTP)—a level of performance that neither of them can attain individually. So far, however, CTP has rarely been observed, suggesting an insufficient understanding of the principle and the application of complementarity. Therefore, we develop a general concept of complementarity and formalize its theoretical potential as well as the actual realized effect in decision-making situations. Moreover, we identify information and capability asymmetry as the two key sources of complementarity. Finally, we illustrate the impact of each source on complementarity potential and effect in two empirical studies. Our work provides researchers with a comprehensive theoretical foundation of human-AI complementarity in decision-making and demonstrates that leveraging these sources constitutes a viable pathway towards designing effective human-AI collaboration, i.e., the realization of CTP.

Introduction. The increasing capabilities of artificial intelligence (AI) have paved the way for collaborating with humans and supporting them in a wide range of domains. Examples include decision support for humans in application areas such as customer services (Vassilakopoulou et al.; 2023), medicine (Jussupow et al.; 2021), law (Mallari et al.; 2020), finance (Day et al.; 2018), and industry (Stauder and Kühl; 2022). With AI decisions becoming increasingly accurate, there is an obvious temptation to fully rely on them and to automate decision tasks. However, this approach often falls short of realizing even better performance by combining and integrating the unique strengths of the individual members in a human-AI team (Seeber et al.; 2020). The recent emergence of large language models illustrates this (Malone et al.; 2023): While applications like ChatGPT often provide helpful, but not always correct results, a human decision-maker can collaborate with the system to, for example, override erroneous responses in order to achieve superior task performance (Malone et al.; 2023). Similarly, in the medical domain, both AI models and physicians are able to produce diagnoses individually. It has, however, been demonstrated that humans and AI models could make different errors (Geirhos et al.; 2021; Steyvers et al.; 2022) so that they may realize superior results when “teamed up”: For instance, the AI model might detect patterns in large amounts of data that humans might not discover easily, while humans might excel at the causal interpretation and intuition required to understand these patterns (Lake et al.; 2017).

This complementarity as the “quality of being different but useful when combined” (Dictionary; 2024) has inspired researchers to investigate how humans’ and AI’s individual abilities could be leveraged to achieve superior team performance compared to either one performing the decision task independently. Such an outcome is defined as complementary team performance (CTP) (Bansal et al.; 2021). This phenomenon is of increasing interest as recent studies tell only part of the story: On the one hand, various studies have demonstrated that human-AI teams are able to outperform human individuals (Alufaisan et al.; 2021; Inkpen et al.; 2023; Liu et al.; 2021; Sarkar et al.; 2023)—often not analyzing, though, whether they also surpass the AI model’s individual performance (Bansal et al.; 2021). On the other hand, while many settings exist in which humans still show better task performance (Grace et al.; 2018, 2024), recent studies also demonstrate evidence for human-AI teams outperforming AI models (Dvijotham et al.; 2023; Fügener et al.; 2022; Ma et al.; 2023).

To make things even more complex, the particular design of human-AI collaboration in decision-making affects humans and their contributions within the team, e.g., the use of unique human knowledge (Fügener et al.; 2021), the self-assessment of human capabilities (Fügener et al.; 2022), the adjustments of mental models (Bauer et al.; 2023), the incentive for “active rethinking” (Lu and Zhang; 2024), or human task performance and learning (Förster et al.; 2024). In summary, current research still misses compelling explanations for the success of human-AI teams as well as a systematic understanding of complementarity when the team performance is measured against the performance of both team members individually. Thus, concentrating on decision-making as an important application of human-AI collaboration (Lai et al.; 2023), we pursue the following two research questions in this work:

RQ1: How can we model human-AI collaboration in decision-making to enable a more nuanced understanding of the synergetic potential in a human-AI team?

RQ2: What factors contribute to complementary team performance?

We address these research questions by developing a conceptualization of human-AI complementarity that introduces and quantifies complementarity potential (CP) and complementarity effect (CE) and outlines the two key sources of complementarity—information and capability asymmetry. In detail, we argue that both complementarity potential and complementarity effect consist of an inherent and a collaborative component. Whereas the first component captures decision-making synergies that, for each task instance, can be attributed to the individually more accurate team member within the human-AI team, the second component captures decision-making synergies that only emerge through collaboration resulting in team decisions which are more accurate than each of the individual ones. We demonstrate the application and the value of our conceptualization in two experimental studies—leveraging the two sources of complementarity, i.e., information and capability asymmetry within the human-AI team. In both studies, humans collaborate with an AI model to conduct decision-making tasks.

Related work. Information asymmetry. Often, humans and AI have access to different sets of information as decision input: On the one hand, AI models are trained on a well-defined, limited, and digitally available set of data (LeCun et al.; 2015). Humans, however, may also have access to information that—due to technical or economic reasons—is not digitized and, thus, not usable for an AI model (Ibrahim et al.; 2021). They may use contextual information or information on rare events for a more holistic decision-making setting. For example, AI models in the medical domain derive diagnoses from X-ray scans by analyzing the pixels of the image (Irvin et al.; 2019). Human radiologists, however, may also draw on information from direct interaction with patients or from access to medical records. On the other hand, the AI may also have access to information not available to the human decision-maker: A driving assistance system may base its actions on sensor data, e.g., from lidar systems that are not accessible by the human while driving (Li and Ibanez-Guzman; 2020).

Capability asymmetry. Even if the information would be identical, different outcomes may be driven by different internal modes of information processing between human and AI (Kühl, Goutier, Baier, Wolff and Martin; 2022). Such capability asymmetries emerge as AI models encode relationships inferred from training data, whereas humans employ compositional mental models that encode beliefs about the physical and social world (Lake et al.; 2017; Rastogi et al.; 2023). While humans are capable of conducting decision-making tasks already after a few trials, AI models require vast amounts of training data to become capable of accurately performing decision-making tasks (Gopnik and Wellman; 2012; Kühl, Goutier, Baier, Wolff and Martin; 2022; Lake et al.; 2017; Tenenbaum et al.; 2011). Similarly, humans have often gained experiences with regard to a particular decision-making task continuously over their lifetime, while AI models’ experiences are limited to the task instances seen during model training and may only be updated from time to time (Dellermann et al.; 2019; Rastogi et al.; 2023). On the other hand, AI may excel in digesting vast amounts of information much faster than humans could, and its processing exhibits a greater capability to perceive even small variations in data (Findling and Wyart; 2021).

Overall, both asymmetries may give rise to performance synergies when collaborating in human-AI teams. A traditional example is forecasting theory (Sanders and Ritzman; 1991, 1995, 2001): Sanders and Ritzman (1995) analyze the effects of combining statistical forecasts with predictions of a human with access to contextual information and find that such combinations could positively impact forecast accuracy.

Method. 3.1 The Principle of Human-AI Complementarity We first motivate the underlying idea of complementarity which drives effective human-AI decision-making. In this work, we focus on the performance on decision-making tasks that humans and AI can conduct independently— recognizing that AI models have elevated above pure decision support for humans. Real-world examples include diagnosing diseases in medicine (Goldenberg et al.; 2019), conducting loan decisions in finance (Turiel and Aste; 2020), and writing entire programs with AI code generation systems (Ross et al.; 2023). However, since neither humans nor AI are perfect, discrepancies in access to information or in capabilities could be leveraged to generate superior outcomes in a human-AI team. Figure 2 illustrates the situation in a simple example for a set of decisions: The decision-making task comprises 25 instances that each have a set of possible discrete outcomes only one of which is correct. The number of incorrect decisions measures the performance of each individual team member and the human-AI team, respectively. The AI makes 13 incorrect decisions, while the human errs at 15 task instances when conducting the task independently. Five of the instances can neither be solved by the AI nor the human on their own. Consequently, if the human-AI team were just to pick the correct decision of either the AI or the human, the team could correctly solve 20 instances and would only miss the 5 that none of them can solve. In other words, relying on the correct individual decisions of each team member improves the result compared to the AI acting alone (as the individually better performing team member): While the AI still shows 13 incorrect decisions when acting independently, teaming up with the human can reduce this number to 5—realizing an improvement potential of 8 decisions. Moreover, it is also conceivable that—while none of the team members is correct individually—the interaction between the team members may allow the generation of correct team decisions even for the remaining 5 red task instances in Figure 2 representing an additional improvement potential2: Team members may recognize through collaboration that they have different information or capabilities that they can jointly apply. Let us, e.g., assume that a task would be to deliver a diagnosis on cancer. Both a human radiologist and an AI analyzing patient data might each individually render a wrong diagnosis, e.g., “malign cancer” conjectured by the human and “no cancer” proposed by the AI. If, however, the human might learn about the AI’s rationale (e.g., regions in the X-ray scan crucial for the decision) or the AI might learn about “side” information that the radiologist has on the patient history, they jointly may arrive at the correct diagnosis “benign cancer”.

For human-AI teams to achieve CTP, it is essential that they manage to realize these improvement potentials. If their information and capabilities could be adequately combined, the team performance in such situations would be superior to their individual performances (Rastogi et al.; 2023).

Differences in human and AI decision-making can arise from information and capability asymmetries as introduced earlier (Section 2.1). These asymmetries are fundamental for the existence of complementarity. They cause different decisions to be taken by humans and AI on an instance level and, therefore, influence the complementarity potential and the realized complementarity effect including both components. Leveraging information asymmetry means making information available to the team: Advantages can be captured as inherent (when the team decision for a task instance corresponds to the individual decision of the team member with “better” information) or as collaborative complementarity potential/effect (when the information of both team members is joined resulting in new insights and a team decision different from the individual ones).

Discussion. Our research emphasizes the importance of complementarity and attaining CTP in human-AI decision-making as an important area of human-AI collaboration (Lai et al.; 2023). In the past, when early AI-based systems still had limited capabilities, CTP was rarely a focus, as these systems could only provide humans with (partial) decision support within a small range of low-stakes decision-making tasks, e.g., calculating decision-making inputs (Power; 2008). Today, however, AI can perform a growing number of tasks independently with human-level performance and is increasingly utilized in high-stakes decision-making domains such as medicine (McKinney et al.; 2020), law (Hillman; 2019), and finance (Day et al.; 2018). It can also provide general-purpose support, driven by recent advances in large language models that enable applications like ChatGPT (Bubeck et al.; 2023). While this opens up the potential for automation, it also enables a new form of collaboration in which AI and humans are equitable team members.

Current research lacks a concise concept of human-AI complementarity, as the majority of studies focusing on human-AI collaboration in decision-making do not achieve CTP (Bansal et al.; 2021; Hemmer et al.; 2021). Moreover, merely observing whether CTP has been achieved or not does not allow in-depth conclusions to be drawn about the synergetic potential of humans and AI through collaboration. In this work, we contribute to a comprehensive understanding of human-AI teams’ inner workings regarding decision-making and provide support on how to achieve CTP more consistently.

5.2 Theoretical Implications Our research’s most important implication is the need to design for CTP, which is influenced by the source of complementarity and the collaboration mechanism to derive a team decision. Both should be taken into account purposefully. Increasing unique knowledge in the team positively affects the inherent complementarity potential. From an AI perspective, this could be achieved by deliberately creating “complementary” AI models designed for teamwork that perform well in areas of the feature space where humans experience difficulties (Hemmer et al.; 2022; Mozannar and Sontag; 2020; Wilder et al.; 2020). From a human perspective, humans could be trained to focus on their unique capabilities and to build awareness to use unique contextual information. Over time, the capabilities of humans and AI may also change. While humans become more experienced, AI models may get updated. For human-AI teams to also function in the long term, it should be ensured that the capabilities of both team members remain complementary to each other over time (Spitzer et al.; 2023), which could be monitored by employing the measures of our conceptualization. Lastly, attaining CTP also implies that the collaboration mechanism should be consciously designed to improve the realization of inherent and collaborative complementarity potential.

Our work also has important implications for managerial decision-makers. In application areas with suitable decisionmaking tasks, responsible managers should focus on deploying AI systems that enable the realization of CTP through collaboration. If they don’t, their competitors could gain advantages. They could start by collecting data to train AI models that are compatible with teamwork and invest in training their employees in order to enhance their human-AI teams’ inherent complementarity potential.

Conclusion. So far, human-AI collaboration in decision-making has been primarily concerned with AI systems helping human users. However, since the number of decision tasks that can be automated (i.e., can be solved by the AI alone) is increasing steadily, the focus has shifted to the purposeful design of the collaboration between humans and AI as team members— thereby shaping the future of work with AI. The ultimate objective of these teams must be to achieve complementary team performance (CTP), with the team outperforming each individual team member. The IS community is predestined to drive the development of appropriate theories and to lay the foundation for practical applications. We hope that the conceptual foundation developed in this paper will provide fruitful ground for future research, and that the empirical studies illustrate the validity and potential of the human-AI complementarity paradigm.

Limitations. 5.4 Limitations Our current research has several limitations that future work needs to address. First, the prevalent use of laboratory-based experiments and vignette methodologies in extant literature on human-AI collaboration in decision-making, including our own, could limit our work’s generalizability and its practical implications. This underscores the need for future studies to use a nuanced approach to prevent contributing to the fragmentation of research in this domain. Furthermore, we focus on developing measurement tools, delineating sources of complementarity, and validating the proposed concepts in behavioral experiments. While our experiments’ controlled settings allow us to derive the insights presented in this work, they do not yet address the wide range of complex human factors, such as motivation (Schunk; 1995), engagement (Benz et al.; 2024; Chandra et al.; 2022), self-efficacy (Westphal et al.; 2024), or behavior patterns within teams (Schecter et al.; 2022), which also contribute to the effectiveness of human-AI teams (Chandra et al.; 2022). In this context, many factors in socio-technical systems are highly interconnected and influence the underlying systems’ acceptance and use (Jain et al.; 2021).