An HCI-Centric Survey and Taxonomy of Human-Generative-AI Interactions

Generative AI (GenAI) has shown remarkable capabilities in generating diverse and realistic content across different formats. HCI literature has investigated how to effectively create collaborations between humans and GenAI systems. However, the current literature lacks a comprehensive framework to better understand human-GenAI interactions, as the holistic aspects of human-centered GenAI systems are rarely analyzed systematically. We present a survey of 291 papers, providing a novel taxonomy and analysis of Human-GenAI Interactions from both human and GenAI perspectives. We highlight challenges and opportunities to guide the design of GenAI systems and interactions toward the future design of human-centered GenAI applications.

Introduction. Recently, Generative Artificial Intelligence (GenAI) models have gained immense popularity and are being applied in diverse applications such as art [10, 184], design [144, 274], and entertainment [183, 297]. Current popular GenAI models including Large Language Models (LLM) and Large Vision Models (LVM) are widely deployed on platforms or in software for their capabilities to create imagery content (Dalle-2 [259], Stable Diffusion [265]), writing literature [344], and Question Answering (ChatGPT [238]). The adoption of GenAI models has demonstrated significant advantages, e.g. fostering creativity [252], driving innovation [39], enabling personalized content generation [323], and providing valuable assistance in creation endeavors [208]. As a result, these models have become ubiquitous, emphasizing the need for well-crafted and compelling interactions between humans and GenAI. To take advantage of the generative power of GenAI models, Human-GenAI interaction techniques such as prompt engineering [195, 252], visualization [308], and interactive interfaces [55, 73], have become popular and effective mediums for humans to interact with GenAI systems. These allow users to collaborate [344], be assisted[8], take suggestions [48] or revise recommendations [304] from GenAI systems.

However, existing research on Human-GenAI interactions focuses on each individual aspect and domain. The key design considerations, common practices, and future research opportunities are still hidden and scattered across many broad topics embedded in GenAI and its applications. To keep pace with the development of GenAI models and their new out-of-the-box capabilities, we identify a need to systematically analyze the research in this field, particularly from an interaction design perspective, to assist the HCI community to innovate and explore new interaction design techniques for the best utilization of GenAI capabilities. Furthermore, this view of GenAI will also foster new emerging applications to consider key vantage points where our framework will point. Inspired by the above, we aimed to lay the groundwork for further developments in the field of human-GenAI interactions by systematically synthesizing all the current research and consolidating the existing knowledge and approaches in this domain.

In this paper, we review a corpus of 291 papers for synthesizing the taxonomy of human- GenAI interactions. Specifically, we synthesize the research fields from both the user and GenAI perspectives (briefly shown in Figure 1) into the following dimensions of the design space: 1) Purposes of Using GenAI, 2) Feedback from Models to Users, 3) Control from Users to Models, 4) Levels of Engagement, 5) Application Domains, and 6) Evaluation Strategies. Our main goal is to present a comprehensive overview of recent developments in and research on AI-model analysis, interaction designs, visualization techniques, and application domains of GenAI-based systems. By compiling the state-of-the-art advancements in these areas, we aim to provide a valuable resource for researchers to understand the current landscape and situate their own work within a broader design space. To achieve the goals above, we summarize a taxonomy from the literature, offering a holistic view that encompasses perspectives from both the GenAI model side (e.g. I/O design, capabilities, and volumes) and the human side (e.g. evaluation strategies and application domains), as well as the interactions between them (e.g. interfaces to control, visualization technique, and feedback design). This taxonomy will enable readers to gain a deeper understanding of the intricacies involved in creating effective and meaningful human-GenAI interaction systems, fostering future evolution and innovation in the design of GenAI technologies. Additionally, we identify open research questions, challenges, and opportunities in the future design of GenAI systems and interactions. By highlighting these areas of exploration, we aim to guide researchers in their pursuit of addressing crucial issues and uncovering new possibilities, so that the HCI community paces and also identifies vantage points for creating new GenAI, with the rapidly evolving technology of GenAI.

Related work. In this section, we cover the developments in GenAI models in prior research as well as open-source platforms and software.

A widely accepted definition [330] of GenAI goes by the probabilistic models that model the joint distribution in contrast to discriminate AI models that model the conditional distribution. From a high-level understanding, GenAI is defined as "an AI that uses existing media to create new, plausible media" [222]. As a sub-topic in the domain of AI, GenAI has a rich history of development. In the early stage of its development, the metaphor of GenAI advanced independently in two major domains, namely Natural Language Processing (NLP) and Computer Vision (CV). In NLP, the generation of nature languages by AI was handled by early implementations of Recurrent Neural Networks(RNN) [70] and Long Short-Term Memory (LSTM) networks [97]. Likewise in CV, the concepts of Artificial/Convolutional Neural Networks (ANN [86] and CNN [170]) were applied to models that generate images. Nevertheless, in both fields, the NN-based methods were greatly limited by the hardware conditions back then. It was not until the early 2010s that the breakthrough in hardware technology enabled the explosion in GenAI research with increased computational power in both NLP and CV fields. Long sentence generation and sequence-to-sequence generation were then achieved by RNNs with much larger sizes and computational power [217, 290]. Similarly in the CV area, models like Generative Adversarial Networks (GAN) [93], Variational Autoencoders (VAE) [159], and their successors [28, 143–145] enabled diverse applications such as style transferring between images [89, 366], generating images based on texts [367], etc. Recent research has highlighted and merged the two fields, enabling the multi-modal generative power of GenAI. Works like Transformer [305] and Diffusion Model [113, 229, 266, 269] have built the theoretical foundation for the current stage of GenAI, where large models such as Generative Pretrained Transformer (GPT) and its successors [31, 149, 254, 255], T5 [256], BERT [62], and CLIP [253] enable diverse applications with content of higher quality and multiple modalities. Empowered by the models, an increasing number of software and interactive platforms are being developed to cater to various fields, such as art, design, education, and algorithm development, democratizing access to the creative potential of GenAI. In this paper, we provide a comprehensive summary of diverse GenAI applications and highlight some notable open-source platforms in Table 15 . These platforms are designed and contributed by researchers, developers, and experts, aiming to make GenAI technology accessible to a broader audience.

Method. As discussed in the former sections, we aim to present a comprehensive taxonomy of human-GenAI interaction in order to help future research and development in designing interactive GenAI systems. Over the collected corpus of literature, we conduct a collaborative analysis to identify the key dimensions of the taxonomy.

The analysis involves multiple stages of review and collaboration among the authors. Initially, each of the authors reads and summarizes a small subset (N= 25) of the papers to establish an approximate taxonomy of components and dimensions. These preliminary taxonomies are then discussed among all authors to iteratively refine and enhance, by highlighting the overlapping dimensions, adding or subtracting components and categories as necessary, and eventually agreeing upon one revised taxonomy. Upon revision, three authors individually read the entire list of papers thoroughly to assign them to their respective categories and dimensions. During this phase, workshop proposals, surveys, and literature reviews are not included in the final categorization but are used as supplementary references to guide our analysis and concretize the design components and categories. To ensure consistency and resolve any conflicts, the three authors subsequently engaged in discussions to finalize the tagging of the papers, by iteratively opening up new discussions upon any author identifying a paper that does not properly fit into the existing taxonomy. At the end of each iteration, authors decide if a new dimension is needed for the taxonomy or agree on the position of the paper in the existing taxonomy. This collaborative approach helps ensure the accuracy and reliability of the categorization process. Eventually, our analysis results in a comprehensive taxonomy of dimensions from an HCI design perspective. The HCI design logic behind this taxonomy suggests that developers and researchers in this field practice User-Centered Design [3] by (1) orienting the system through the needs of the end-users (section 4) (2) considering the capabilities and functions of the GenAI models (section 5) (3) exploring the possible actions of the users, the corresponding results of the actions, and the mapping between the actions and the results(section 6) (4) enabling different levels of engagement in the interaction for different tasks (section 7) (5) deciding the application domain the system can be generalized into (section 8), (6) and evaluating the system via designated metrics (section 9). In the following sections, we discuss various components in the dimensions and the motivations for considering them when designing a system. Later on, we present the future opportunities and challenges in this domain summarized from our literature review. In the appendix, we included tables containing all the papers’ citations and counts that fall into respective categories and dimensions.

4 PURPOSES OF USING GENAI The initial step of designing an interactive GenAI system is to determine the end goals of (i.e. purposes of using) the system from the user’s perspective. Characterizing such purposes sets the fundamental notes on the design of the systems and answers the essential questions on what types of tasks the system is to fulfill and what the users of the system expect from it. To this end, in the very first section, we categorize the purposes of users of GenAI applications. On a high level, we identify the purposes falling into the following categories: 1) Refine the Outcome, 2) Explore Alternatives, 3) Get Answers for Inquiries, 4) Understand a Subject, 5) Automate Processes, 6) Enhance Experiences, and 7) Augment Sample Data.

Purpose-1 Refine the Outcome. Users possess certain expectations of an instance or outcome to be created. With a specific objective, users utilize GenAI applications to generate instances to meet their qualitative or quantitative expectations.

Discussion. 10 FINDINGS Based on the analysis of our taxonomy, Figure 10 shows a summary of the number of papers for each dimension In this section, we discuss the standard strategies and gaps that we identify through our analysis of the literature.

Finding-1 Mediums of Control: Direct but not Intuitive. Through our literature review, we notice that direct control modalities are more applied (e.g. widgets, controllers, drawings and highlighting, and text, N= 234) than the intuitive ones (e.g. gestures, brain signals, and voice, N= 49) to control the output of the GenAI system. Direct control modalities allow users to modify the attributes of the models or data straightforwardly, while intuitive control modalities require mappings from the users’ intuition to the functions of the models. This inclination highlights that GenAI systems align more with the need to tweak the GenAI models for specific functions directly while overlooking the users’ need for intuitive interactions. For example, using sliders to adjust the weights of the attributes of a GAN model [72, 73, 308] is direct yet not intuitive, because novice users (who do not know AI) do not possess the technical knowledge to understand the correspondences between the attributes and the outputs. When users interact with systems with straightforward interactions, they need to build the mapping between their interaction input and the output [3], while with intuitive interactions, researchers have preset this mapping for the users. Put simply, considering the Gen-AI systems as black boxes with unknown I/O correspondence, intuitive interactions foster smoother learning curves of this I/O correspondence than straightforward interactions. This observation suggests that intuitive human interactions are not yet the mainstream mediums for controlling the GenAI models.

Finding-2 Visualizing the Results rather than the Process. We notice that most GenAI systems do not reveal the intermediate layers or output to the users (N= 231). This is hard to accomplish from the AI-developing side, given the fact that it is hard for end-users to comprehend the mathematical functions that lie within the intermediate layers of the users. However, from an HCI perspective, we highlight the necessity of investigating the design space of interacting with the intermediate layers of the GenAI systems, which is a significant missing piece in the current research. It is important that users understand the intention-action mapping [3, 232] of using a system, i.e. what consequences result from each of their interactions. From the designer’s perspective, it is required to understand the complexity of the models so that designers can "conceptualize the system’s behaviors in order to choreograph its interactions" [338]. From the user’s perspective, users need to interpret outputs and express feedback to the models [68], which can benefit from a multi-layer design of the system compared to an end-to-end one. Starting from this consideration, we further discuss the future directions to address this concern in section 11 Finding-3 the Use of Foundation Models. We observed that a major GenAI utilized in the research is Large Language Models (LLMs, a significant subset of Foundation Models, which are defined as models (e.g., BERT, DALL-E, GPT-3) trained on broad data (generally using selfsupervision at scale) that can be adapted to a wide range of downstream tasks [27, 160, 219] in 82 papers). Large language models have gained significant popularity due to their unparalleled ability to understand and produce human-like text. We also observed that most LLM-based applications utilize textual conversation as their interaction modality, implying that the users of these LLM-based systems interact by text input.

Conclusion.