When combinations of humans and AI are useful: A systematic review and meta-analysis

Inspired by the increasing use of AI to augment humans, researchers have studied human-AI systems involving different tasks, systems, and populations. Despite such a large body of work, we lack a broad conceptual understanding of when combinations of humans and AI are better than either alone. Here, we addressed this question by conducting a pre-registered systematic review and meta-analysis of over 100 recent experimental studies reporting over 300 effect sizes. First, we found that, on average, human-AI combinations performed significantly worse than the best of humans or AI alone (Hedges’ g = −0.23, 95% confidence interval −0.39 to −0.07). Second, we found performance losses in tasks that involved making decisions and significantly greater gains in tasks that involved creating content. Finally, when humans outperformed AI alone, we found performance gains in the combination, but when the AI outperformed humans alone we found losses. These findings highlight the heterogeneity of the effects of human-AI collaboration and point to promising avenues for improving human-AI systems.

Introduction. People increasingly work with artificial intelligence (AI) tools in fields including medicine, finance, and law, as well as in daily activities such as traveling, shopping, and communicating. These human-AI systems have tremendous potential given the complementary nature of humans and AI – the general intelligence of humans allows us to reason about diverse problems, and the computational power of AI systems allows them to accomplish specific tasks that people find difficult.

In fact, a large body of work suggests that integrating human creativity, intuition, and contextual understanding with AI’s speed, scalability, and analytical power can lead to innovative solutions and improved decision-making in areas such as healthcare [1], customer service [2], and scientific research [3]. On the other hand, a growing number of studies reveal that human-AI systems do not necessarily achieve better results when compared to the best of humans or AI alone. Challenges such as communication barriers, trust issues, ethical concerns, and the need for effective coordination between humans and AI systems can hinder the collaborative process [4–9].

These seemingly contradictory results raise important questions: When do humans and AI complement each other? And by how much? To address these issues, we conducted a systematic literature review and meta-analysis in which we quantified synergy in human-AI systems and identified factors that explain its presence (or absence) in different settings.

We focused on two outcomes: (1) human-AI synergy, where the human-AI group performs better than both the human alone and the AI alone, which is analogous to strong synergy in human groups [10, 11]; and (2) human augmentation, where the human-AI group performs better than the human alone (see Section S1.1 in the SI for more details).

When evaluating human-AI systems, many studies focus on human augmentation [12–15]. This measure can serve important purposes in contexts for which full automation cannot happen for legal, ethical, or safety reasons and in cases when AI does not align with human values. But when talking about the potential of human-AI systems, most people implicitly assume that the combined system should be better than either alone; otherwise, they would just use the best of the two [16]. Thus they are looking for human-AI synergy. In light of these considerations, a growing body of work emphasizes evaluating and searching for synergy in human-AI systems [4, 8, 17–20].

Method. We focused on two outcomes: (1) human-AI synergy, where the human-AI group performs better than both the human alone and the AI alone, which is analogous to strong synergy in human groups [10, 11]; and (2) human augmentation, where the human-AI group performs better than the human alone (see Section S1.1 in the SI for more details).

When evaluating human-AI systems, many studies focus on human augmentation [12–15]. This measure can serve important purposes in contexts for which full automation cannot happen for legal, ethical, or safety reasons and in cases when AI does not align with human values. But when talking about the potential of human-AI systems, most people implicitly assume that the combined system should be better than either alone; otherwise, they would just use the best of the two [16]. Thus they are looking for human-AI synergy. In light of these considerations, a growing body of work emphasizes evaluating and searching for synergy in human-AI systems [4, 8, 17–20].

Discussion. Systems that combine human intelligence and AI tools can address multiple issues of societal importance, from how we diagnose disease to how we design complex systems [22–24]. But some studies show how augmenting humans with AI can lead to better outcomes than humans or AI working alone [24–26], while others show the opposite [4, 7, 9]. These seemingly disparate results raise two important questions: How effective is human-AI collaboration, in general? And under what circumstances does this collaboration lead to performance gains versus losses? Our study analyzes over three years of recent research to provide insights into both of these questions.

Performance Losses from Human-AI Collaboration: Regarding the first question, we found that, on average among recent experiments, human-AI systems did not exhibit synergy: the human- AI groups performed worse than either the human alone or the AI alone. This result complements the qualitative literature reviews on human-AI collaboration [27–29], which highlight some of the surprising challenges that arise when integrating human and artificial intelligence. For example, people often rely too much on AI systems ("overreliance"), using its suggestions as strong guidelines without seeking and processing more information [6, 30, 31]. Other times, however, humans rely too little on AI ("underreliance"), ignoring its suggestions because of adverse attitudes toward automation [7, 31, 32].

Interestingly, we found that, among this same set of experiments, human augmentation did exist in the human-AI systems: the human-AI groups performed better than the humans working alone.

Thus, even though, on average, the human-AI combinations did not achieve synergy, the AI system did, on average, help humans perform better. This result can occur, of course, because by definition the baseline for human-AI synergy is more stringent than that for human augmentation. It may also occur, however, because obtaining human-AI synergy requires different forms of human-AI interaction, or because the recent empirical studies were not appropriately designed to elicit human-AI synergy.

Moderating Effect of Task Type: With the large data set we collected, we also performed additional analyses of factors that influence the effectiveness of human-AI collaboration. We found that the type of task significantly moderated synergy in human-AI systems with decision tasks associated with performance losses and creation tasks associated with performance gains.

We hypothesize that this advantage for creation tasks occurs because even when creation tasks require the use of creativity, knowledge, or insight for which humans perform better, they often also involve substantial amounts of somewhat routine generation of additional content that AI can perform as well or better than humans. For instance, generating a good artistic image usually requires some creative inspiration about what the image should look like, but it also often requires a fair amount of more routine fleshing out of the details of the image. Similarly, generating many kinds of text documents often requires knowledge or insight that humans have and computers do not, but it also often requires filling in boilerplate or routine parts of the text as well.

With most of the decision tasks studied in our sample, however, both the human and the AI system make a complete decision, with the humans usually making the final choice. Our results suggest that with these decision tasks, better results might have been obtained if the experimenters had designed processes in which the AI systems did only the parts of the task for which they were clearly better than humans.

Conclusion. In conclusion, our results demonstrate that human-AI systems often perform worse than humans alone or AI alone. But our analysis also suggests promising directions for the future development of more effective human-AI systems. We hope that this work will help guide progress in developing such systems and using them to solve some of our most important problems in business, science, and society.

Limitations. Importantly, we want to highlight some general limitations of our meta-analytic approach to aid with the interpretation of our results. First, our quantitative results apply to the subset of studies we collected through our systematic literature review. To evaluate human-AI synergy, we required that papers report the performance of (1) the human alone, (2) the AI alone, and (3) the human-AI system. We can, however, imagine tasks that a human and / or AI cannot perform alone but can when working with the other. Our analysis does not include such studies.

Second, we calculate effect sizes that correspond to different quantitative measures such as task accuracy, error, and quality. By computing Hedges’ g, a unitless standardized effect size, we can describe important relations among these experiments in ways that make them comparable across different study designs with different outcome variables [36]. The studies in our data set, though, come from different samples of people – some look at doctors [37–39], others at crowdworkers [4, 6, 34], and still others at students [13, 40, 41] — and this variation can limit the comparability of the effect size to a degree [36]. Additionally, the measurement error can also vary across experiments.