Ironies of Generative AI: Understanding and mitigating productivity loss in human-AI interactions

Ironies of Generative AI: Understanding and mitigating productivity loss in human-AI interactions AUSTE SIMKUTE∗, University of Edinburgh, United Kingdom LEV TANKELEVITCH∗, Microsoft Research, United Kingdom VIKTOR KEWENIG, University College London, United Kingdom AVA ELIZABETH SCOTT, University College London, United Kingdom ABIGAIL SELLEN, Microsoft Research, United Kingdom SEAN RINTEL, Microsoft Research, United Kingdom Generative AI (GenAI) systems offer opportunities to increase user productivity in many tasks, such as programming and writing.

However, while they boost productivity in some studies, many others show that users are working ineffectively with GenAI systems and losing productivity. Despite the apparent novelty of these usability challenges, these ‘ironies of automation’ have been observed for over three decades in Human Factors research on the introduction of automation in domains such as aviation, automated driving, and intelligence. We draw on this extensive research alongside recent GenAI user studies to outline four key reasons for productivity loss with GenAI systems: a shift in users’ roles from production to evaluation, unhelpful restructuring of workflows, interruptions, and a tendency for automation to make easy tasks easier and hard tasks harder. We then suggest how Human Factors research can also inform GenAI system design to mitigate productivity loss by using approaches such as continuous feedback, system personalization, ecological interface design, task stabilization, and clear task allocation. Thus, we ground developments in GenAI system usability in decades of Human Factors research, ensuring that the design of human-AI interactions in this rapidly moving field learns from history instead of repeating it.

Introduction. Generative artificial intelligence (GenAI) systems, such as large language models (LLMs) that can generate novel content and perform many other tasks, present myriad opportunities and challenges to humans in knowledge-intensive domains. GenAI applications have emerged in domains such as healthcare [87], research [72], writing [18, 29], creative ∗Both authors contributed equally to this research.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from permissions@acm.org.

© 2024 Copyright held by the owner/author(s). Publication rights licensed to ACM. Manuscript submitted to ACM arXiv:2402.11364v1 [cs.HC] 17 Feb 2024 –, 2024, – Simkute and Tankelevitch, et al. work [46, 65, 89, 96], consulting [31], and recruitment [14]. Software engineering has been particularly impacted, with GenAI-assisted programming tools, such as GitHub Copilot [43], being increasingly used to support software engineering practices and perform tasks such as auto-completing code [62], translating code across languages, and answering programming questions, among others [105, 109].

GenAI’s ability to solve domain-specific problems speaks to its potential to augment human performance and transform productivity. Recent research already suggests the enormous positive impact these systems could have on workers’ performance in domains including programming [99], writing [88], law [20], and consulting [31]. Based on this research, the expectation is that new tools will often free up users’ time and allow them to focus on higher-level tasks, increasing their productivity. However, when using the new tools in practice, many users, such as programmers, report increased cognitive load, frustration, and time spent on the tasks that GenAI is intended to support. Feedback from Copilot users, as well as usability studies of GenAI-driven programming tools, suggest that, in some cases, using GenAI support can, in fact, lead to productivity loss. For example, software engineers and novice programmers struggle to effectively prompt systems, debug generated code, lose their state of flow when interrupted by long code suggestions, and get stuck in ineffective practices, such as reviewing, editing and then ultimately deleting suggestions [8, 100, 109].

Similar observations are emerging in creative domains, where graphic [65, 89] and manufacturing [46] designers struggle with prompt engineering and other aspects of GenAI interaction. This suggests that the potential of GenAI systems to boost productivity may not be guaranteed, evenly distributed, or fully exploited.

These observations mirror the long line of Human Factors studies exploring human-automation interactions in safetycritical systems in aviation, industrial plants, and other areas [36, 66]. Indeed, they reflect the ‘ironies of automation’ [7], which capture the idea that the more advanced an automated system is, the more important the human operator may be.1 Despite automation taking over human manual control in areas where it is expected to provide superior performance, humans are still left to supervise automation. However, operators might have insufficient support to supervise, and so instead of being supported by automation, they find themselves cognitively overburdened, trying to decipher systems’ outputs and spot errors. Similarly, in the context of GenAI, users’ roles have shifted from producing output to evaluating it, often with little contextual information and situational awareness. This is exacerbated by GenAI tools’ ability to produce outputs at a capacity too demanding for adequate evaluation, with questionable reliability, and with poor explainability [17, 69, 110]. Moreover, poor system and interface design can result in unhelpful restructuring of workflows, which increases cognitive load and undermines productivity gains [7]. This is echoed in programmers’ experiences and feedback around Copilot features [8, 100, 109], with evidence of similar effects emerging in other domains [29, 46, 49]. Finally, as a result of which tasks get automated, as well as poor system design, automation often makes easy tasks easier while making hard tasks even harder. This same pattern is now being observed in usability studies of GenAI systems [8, 109].

Related work. These observations mirror the long line of Human Factors studies exploring human-automation interactions in safetycritical systems in aviation, industrial plants, and other areas [36, 66]. Indeed, they reflect the ‘ironies of automation’ [7], which capture the idea that the more advanced an automated system is, the more important the human operator may be.1 Despite automation taking over human manual control in areas where it is expected to provide superior performance, humans are still left to supervise automation. However, operators might have insufficient support to supervise, and so instead of being supported by automation, they find themselves cognitively overburdened, trying to decipher systems’ outputs and spot errors. Similarly, in the context of GenAI, users’ roles have shifted from producing output to evaluating it, often with little contextual information and situational awareness. This is exacerbated by GenAI tools’ ability to produce outputs at a capacity too demanding for adequate evaluation, with questionable reliability, and with poor explainability [17, 69, 110]. Moreover, poor system and interface design can result in unhelpful restructuring of workflows, which increases cognitive load and undermines productivity gains [7]. This is echoed in programmers’ experiences and feedback around Copilot features [8, 100, 109], with evidence of similar effects emerging in other domains [29, 46, 49]. Finally, as a result of which tasks get automated, as well as poor system design, automation often makes easy tasks easier while making hard tasks even harder. This same pattern is now being observed in usability studies of GenAI systems [8, 109].

Method. 2 PRODUCTIVITY CHALLENGES OF GENERATIVE AI AUTOMATION Here, we outline the key productivity challenges that have been observed in human-automation interaction over decades of Human Factors research and are now becoming apparent in user studies of GenAI systems. Our focus is on GenAI systems, the integrated whole comprising GenAI models and interfaces. Some challenges pertain to GenAI models (e.g., issues around prompting), and some pertain to interface design (e.g., issues around task interruptions).

We begin with challenges related to the shift from manual control or production to a more passive supervisory role of the user, such as monitoring and evaluation of AI outputs (Section 2.1). We explore specific aspects related to this shift, such as reduced situational awareness, the contributory factors of automation’s high capacity, complexity and opaqueness, reliability, and potential resultant complacency and over-reliance. We then outline how the introduction of automation such as GenAI can unhelpfully restructure users’ workflows, stifling their productivity (Section 2.2). We focus on how the introduction of new tasks, such as prompting or output adaptation, can affect user performance and how workflow restructuring can lead to loss of task sequence and feedback. We also explore the influence that task interruptions from AI suggestions can have on users’ productivity (Section 2.3). Finally, we explore how automation such as GenAI can paradoxically lead to easy tasks being made easier and hard tasks made harder, a phenomenon we refer to as ‘task-complexity polarization’ (also known as "clumsy automation" in Human Factors research [125]; Section 2.4). Figure 1 outlines the four types of challenges.

Discussion. 3 HUMAN FACTORS SOLUTIONS Beyond diagnosing the usability challenges of automation, Human Factors research has spent decades studying approaches to mitigate these challenges (e.g., [36, 91, 93, 112]). Here, we outline some key potential design solutions that could reduce the productivity loss in human-GenAI interaction. These include providing continuous relevant feedback to users (Section 3.1), enabling system personalization (Section 3.2), applying ecological interface design (Section 3.3), using task stabilization and interruption timing techniques (Section 3.4), and enabling clear task allocation between users and systems (Section 3.5). Besides targeting individual productivity loss challenges, these solutions share the underlying Human Factors principles of providing feedback and enabling system flexibility [16].

More broadly, we argue that these proposed approaches aim to (i) increase user agency in how they adapt the GenAI support to users’ preferred ways of working, reducing the cognitive load stemming from disrupted workflows; (ii) increase users’ situational awareness of system changes and potential errors, reducing the cognitive load associated with the monitoring and evaluation of AI outputs; and (iii) increase user flexibility through the more granular application of AI support to their tasks, freeing users from having to make a binary decision of either using GenAI tools potentially ineffectively or not using them at all [17, 109]. Throughout, we focus on the programming domain as an example of how these approaches can be applied to GenAI systems.

3.1 Continuous feedback When GenAI is introduced to users’ workflows, their role can shift from active involvement in performing the task (i.e., production) to more passively reviewing the AI-generated outputs for errors (i.e., output evaluation). The latter is a cognitively demanding task due to the lack of supporting contextual information and the resultant loss of situational awareness. We propose that feedback about system behavior is a key strategy to keep users engaged and in the loop of GenAI system performance.

During the monitoring stages, receiving continuous feedback is crucial for the operator to remain in the loop and recognise moments when interruption and input are needed [66, 71]. Feedback is essential to help operators know if their requests have been received if the actions of the automation system are being performed properly, and if any Ironies of Generative AI –, 2024, – errors are occurring [86]. With GenAI systems, this includes knowing which aspects of the input are serving as prompts, how they are being interpreted by the system, how the output matches them, and whether there are any errors. Thus, feedback is tied to carefully designed explainability features [69, 116]. It should help users know why the system is responding in a certain way and allow them to build mental models of the system’s behaviour, how it interacts with them, and where they can expect failures (i.e., cause-and-effect relationships). Moreover, by helping users develop a more accurate mental model of the system, feedback can also serve to support users in better prompting and output adaptation [17, 69], thereby helping them structure their workflows more effectively.

We suggest that GenAI tools should continuously provide relevant feedback to users, updating them on the system’s state, particularly during the monitoring stages. Feedback should be informative but non-intrusive, where the amount and form of feedback adapts to the interactive style of the participants and the nature of the problem [86].

Conclusion. We have synthesized and analyzed the productivity challenges emerging during human-GenAI interactions, focusing on the much-studied domain of software development and noting similarities in areas such as data science, design, and writing. We have demonstrated the parallels between productivity challenges in older Human Factors automation studies and recent GenAI studies. Drawing on the human automation studies, we have categorised these challenges and the underlying reasons related to Human Factors, such as workload, feedback, and situational awareness. We show how aspects like the shift from active production (e.g., writing code) to passive evaluation (e.g., reviewing code), unhelpful workflow restructuring, task interruptions, and task-complexity polarization can stifle human performance and effective implementation of GenAI.

Further extrapolating from human-automation studies, we have provided a set of design solutions that could help avoid productivity losses in human-GenAI interaction. More broadly, we argue for more consideration of users’ workflows, unique ways of working, and domain specificities when designing GenAI tools. To achieve this, we propose that systems be designed in accordance with ecological interface design, the principle of continuous feedback, support for flexibility via task allocation between users and systems, and user-guided system personalization. We also provide concrete design solutions for effectively guiding user attention during interruptions.

Our paper is an initial bridge between Human Factors and Human-Computer Interaction issues of human-GenAI interaction. There is, of course, far more nuanced Human Factors research that can help understand and address the key productivity challenges in this fast-paced area. Reciprocally, we also expect that future Human-Computer Interaction research may open up new domains of exploration for Human Factors.