The Participatory Turn in AI Design: Theoretical Foundations and the Current State of Practice

Despite the growing consensus that stakeholders affected by AI systems should participate in their design, enormous variation and implicit disagreements exist among current approaches. For researchers and practitioners who are interested in taking a participatory approach to AI design and development, it remains challenging to assess the extent to which any participatory approach grants substantive agency to stakeholders. This article thus aims to ground what we dub the “participatory turn” in AI design by synthesizing existing theoretical literature on participation and through empirical investigation and critique of its current practices. Specifically, we derive a conceptual framework through synthesis of literature across technology design, political theory, and the social sciences that researchers and practitioners can leverage to evaluate approaches to participation in AI design. Additionally, we articulate empirical findings concerning the current state of participatory practice in AI design based on an analysis of recently published research and semi-structured interviews with 12 AI researchers and practitioners. We use these empirical findings to understand the current state of participatory practice and subsequently provide guidance to better align participatory goals and methods in a way that accounts for practical constraints.

Introduction. As artificial intelligence (AI) systems are developed and deployed across various sectors (e.g., hiring, healthcare, education, content moderation), there have been increasing calls to involve members of communities impacted by AI systems in their design [7, 99, 121, 129, 136, 137, 162, 178, 179, 213, 221]. In part, such calls for participation in AI design argue that participation can enable AI systems to better reflect the values, preferences, and needs of users and other impacted stakeholders, or more broadly, that participation will empower stakeholders in shaping the design of AI systems [3, 7, 19, 26, 33, 51, 61, 118, 137, 143, 172, 189, 198–200]. However, despite a growing consensus that stakeholders should participate more in AI design, there is enormous variation in the methods and theories applied to achieve that participation, even with respect to the goals for leveraging participation in the first place. There are, for example, practitioners that take a communitycentric approach in defining and operationalizing stakeholder participation upstream of any system or product design activity [110]; community collectives of African NLP researchers [156] and queer AI researchers [175] have formed to design and evaluate algorithmic systems that reflect their communities’ needs. In a commercial context, industry practitioners argue that participation helps develop products that better align with people’s different wants and needs and that it accelerates innovation [193] while improving profitability [88]. Further downstream from a traditional design phase, many practitioners involve stakeholders as the “human infrastructure” [144] or “data labor” [146] that enables ML systems to work. This involves including stakeholders in data generation and interactions with data collection infrastructures [17, 59, 144], data labeling or annotation [103, 147, 169, 209], feedback for optimization [15, 46, 81, 135], and end users’ discretionary work to make the system work [5]. Meanwhile, some researchers have begun to leverage language models to generate synthetic user research data positing such data as suitable “proxies for understanding human experiences” [10, 91].

As large-scale AI models such as large language models are trained on ever larger swaths of human-generated data [e.g. 63] and “fine-tuned” through processes involving human feedback in some form [e.g., 15], the stakes are high for what it means for stakeholders to meaningfully participate in shaping the design of AI—and what it might mean for an AI system or its design process to be branded as “participatory AI,” with all of the implications for empowering stakeholders that that term implicitly conveys. While diversity in approach is warranted to address the various opportunities and constraints present in any system design context, the current state of participatory AI speaks to a largely unaccounted for and unaccountable heterogeneity that lacks a shared ethos or set of principles. To address this gap, we attempt in this paper to provide an analytic structure and empirical foundation to help mature participatory practice in AI design. To do that, we first synthesize theories and approaches to participation across multiple fields (e.g., technology design, policy and governance, international development) to develop a conceptual framework that enables us to articulate key distinctions across a broad spectrum of participatory strategies and tactics. We then use this conceptual framework to analyze a corpus of 80 research papers that describe their approach as “participatory AI.” In mapping this corpus to our conceptual framework, we map out the contemporary terrain of practice. Further, we interviewed 12 authors of papers in this corpus to understand the motivations, challenges, and aspirations of researchers leveraging participatory AI. We find that most current participatory AI efforts consult stakeholders for input on discrete implementation parameters, rather than empower them to make key AI design decisions. In addition, we find that researchers and practitioners implementing participatory AI projects experience a tension between their aspirations to empower participants and the practical constraints (e.g., resources, timelines) they faced in attempting to do so. To resolve these tensions, we find that participatory AI projects adopt tactics such as relying on proxies for participation—including both human and algorithmic proxies—to represent stakeholders in shaping AI systems. Such tactics raise important questions regarding the affordances and limits of simulated stand-ins in the context of AI design. This paper makes three contributions. First, we provide a conceptual framework for analyzing participatory approaches to AI design in a way that connects goals to techniques. Second, we contribute empirical findings about motivations and challenges within the current paradigm of participatory AI.

Related work. 2 STAGE ONE: ESTABLISHING A FRAMEWORK FOR EVALUATING PARTICIPATION 2.1 Related Work Given the heterogeneity of participatory AI, it remains challenging to have substantive discussions about the extent to which a particular participatory approach achieves one’s goals (e.g., for inclusiveness or empowerment) [e.g., 12, 80]. To this end, the first goal of this paper is to develop a conceptual framework as a point of reference that can provide theoretical grounding for such evaluations. Recent scholarship has started working toward typologies for classifying the different modes of participation in AI. Sloane et al. [203] elucidates the distinction between participation as work, as consultation, and as justice; Birhane et al. [26] identifies three objectives of participatory AI: for algorithmic performance improvements, process improvements, and collective exploration; the Ada Lovelace Institute [104] proposes a “spectrum of participation” to describe five levels of participatory mechanisms, from inform, consult, involve, collaborate, to empower; similarly, Berditchevskaia et al. [24] conceptualizes four levels of participatory AI: consultation, contribution, collaboration, and co-creation. While these are valuable starting points for conceptualizing and defining participatory AI, as Birhane et al. [26] points out, “there are yet no clear consensus on what minimum set of standards or dimensions one should use to assess or evaluate a given potential participatory mechanism.”

Method. 2.2 Participatory Traditions Across Disciplines To bring some structure to our analysis of the current heterogeneous state of participatory AI, we took a step back from the contemporary AI literature and broadly reviewed existing literature on participatory traditions from multiple disciplines, including: technology design [e.g., 153, 201], political theory [e.g., 12, 80, 173], and the social sciences [e.g., 212]. We pulled from works that specifically addressed the nature of participation in policy and design processes, as well as works cited by more recent participatory AI papers (e.g., [196], as invoked by [126]). In addition to foundational theoretical and methodological works in these traditions, we included literature that offered critiques and challenges to participatory practices in their respective fields [e.g., 54]. The specific traditions we drew from for our synthesis include user-centered design, service design, participatory design, co-design, and value-sensitive design, as well as participatory action research, participatory democracy (including deliberation theory), social choice theory, and mechanism design. Below we provide a high-level overview of each of these traditions and their corresponding tenets.

2.3 Approaches to Stakeholder Participation • User-centered design (UCD): involving end users during needfinding and evaluation of designs. The double-diamond UCD process has become the most common design process in industry practice [57]. Technology designers engage end users in identifying user needs and assessing design ideas [142, 161]. • Service design: involving direct and indirect stakeholders impacted by a technology service in its design. Using methods such as service blueprints and value stream mapping, service designers consider how their design decisions impact stakeholders [74, 74, 75, 100, 182]. • Participatory design (PD): incorporating diverse voices in design in order to challenge power structures. In contrast with UCD and service design, PD is designed to enable non-expert stakeholders to provide direct input on technology design [86, 201]. PD provides guidance for practitioners to create a hybrid space (i.e., neither in the stakeholders or the designers’ domain) that allows diverse non-expert stakeholders to meaningfully contribute, while uncovering and challenging the power dynamics between stakeholders and designers, as well as within groups of stakeholders [93, 122, 153]. • Co-design: creative cooperation between stakeholders and technology designers across the whole span of a design process [116, 185, 206]. While closely related to PD, co-design typically lacks its explicitly political component [206]. Further, co-design often focuses more explicitly on designing specific products or artifacts [116], while PD may focus more on the larger sociotechnical system in which particular artifacts will be embedded [201]. • Value-sensitive design (VSD): accounting for the values of direct and indirect stakeholders. VSD is similar to PD, but focuses on identifying and incorporating the values of direct and indirect stakeholders in technology design [78, 79]. • Participatory action research (PAR): engaging stakeholders as co-inquirers to co-construct research plans and interventions [68, 94, 176, 212, 218]. Originating in the social sciences (and adopted in HCI [93, 94]), PAR involves stakeholders collaborating with researchers to develop research questions and study designs, conduct research with researchers, and interpret results [68, 94, 176, 212, 218], focusing on research rather than design (as in PD) [84]. • Social choice theory (SCT) and mechanism design: quantitative aggregation of stakeholder preferences. SCT focuses on identifying people’s preferences for (e.g.) public policies and mathematically aggregating those in a preference-ranking model [e.g., 13, 108, 184, 195, 196]. Prior work in the field of mechanism design has used SCT to develop a framework for including stakeholders in algorithmic decision-making [1, 71, 98, 215]. • Participatory democracy and civic participation: involving citizens and stakeholders in civic decision-making.

Discussion. 3.2.1 The majority of projects operationalized stakeholder participation via discrete preference or value elicitation. Our analysis of the corpus of research articles indicates that the current state of participatory AI design falls largely under the consultative mode in which preferences and values are elicited from stakeholders through a variety of techniques. The output of these elicitation processes is then analyzed by AI researchers in a broader loop of design and development activities.

From our corpus analysis, we found that in motivating “why is participation needed”, the majority of projects described their goal as to improve the user experience (80 out of 80) or to better align AI with stakeholders’ preferences and values (52 out of 80). This reflects a state of participatory practice which focuses less on collaboration and stakeholder ownership and more on consultation and inclusion. In fact, only 10% (8 out of 80) of the projects involved stakeholders with the goal of shaping the system’s scope and purpose. See Figure 2, first row. In terms of the scope of participation, the majority of projects surveyed focused their participatory interventions exclusively on informing selected aspects of the user interface, thus restricting “what is on the table?” to a particular component of the system in question (and notably one that does not directly link to the algorithmic or machine learning aspects of the system). In fact, all projects involved stakeholders in shaping the user interface (UI) design of the AI system, such as designing the explanations around AI outputs. Shaping the design of AI models is rarely on the table for stakeholders. Only 10% (8 of 80) of the projects allowed stakeholders to provide input into the type of model and features used, the design of objective or loss functions, or (for AI classification models) the decision thresholds. None of the projects allow stakeholders to rule out AI as a solution, since these projects were defined as “AI projects” in the first place. Mapping these observations onto the levels of participation framework, we can see that a vast majority of projects fall into the consultative realm in terms of “what’s on the table“. See Figure 2, second row. As it relates to “who is involved?,” the majority of stakeholders brought onto the projects in our corpus were identified and chosen by project leads (74 of 80). Often (47 of 80), project leads’ choices were based upon what they presumed to be the background of that group/person, recruiting stakeholders either opportunistically (“just talk to some of your friends” [P10]) or through an opt-in mechanism (“Some people just kind of came to (our) team and said, hey, can we work with you?” [P9]. This contrasts with the much smaller number of projects that leveraged a community-engaged approach in establishing relationships with stakeholder groups and leveraging those relationships to help designate participants for design activities (6 out of 80). Even fewer projects (3 out of 80) had community stakeholders involved throughout the life cycle of the AI design project. Overall, reliance on project leads’ criteria for deciding “who’s involved” across the majority of projects means less agency is given to community stakeholders in terms of design decision-making. Further, the relative scarcity of relationship-building and stakeholder involvement throughout the end-to-end AI lifecycle points more toward an ad-hoc mode of participation of current practice. See Figure 2, third row. As it relates to participatory methods, 85% (68 out of 80) of the projects engaged stakeholders in eliciting preferences or values through methods such as surveys and interviews, role-playing, and collaborative story-boarding.

Conclusion. The Participatory Turn in AI Design EAAMO ’23, October 30–November 01, 2023, Boston, MA participatory goals and implement participatory methods on their projects.

Limitations. 4.4 Limitations Although we developed our corpus of research articles from a range of publication venues, using a variety of keywords, we may have missed out on relevant projects in venues outside of those we included here (e.g., work published outside of computing conference venues, including HCI journals and ‘grey literature’ such as white papers from policy institutes), and may have missed projects that used different terms from those we searched for to describe the participatory work they did. In addition, although we conducted a purposive sampling of authors of research articles in the corpus to recruit for the interviews, there may be key perspectives of researchers and practitioners from the corpus who we may have failed to reach, or who declined to participate. Finally, our interviews captured the perspectives, processes, and challenges expressed by researchers and practitioners working on participation in AI. Future work should incorporate perspectives from other key stakeholders, such as policymakers or others at civil society or community-based organizations, and, most crucially of all, people who have participated in designing any of the AI systems in the corpus, or stakeholders who are or may be impacted by particular AI systems more generally, to understand whether and how they might want to participate in shaping the design of the systems that impact their lives.