Red Teaming LLMs as Socio-Technical Practice: From Exploration and Data Creation to Evaluation

Recently, red teaming, with roots in security, has become a key evaluative approach to ensure the safety and reliability of Generative Artificial Intelligence. However, most existing work emphasizes technical benchmarks and attack success rates, leaving the socio-technical practices of how red teaming datasets are defined, created, and evaluated under-examined. Drawing on 22 interviews with practitioners who design and evaluate red teaming datasets, we examine the data practices and standards that underpin this work. Because adversarial datasets determine the scope and accuracy of model evaluations, they are critical artifacts for assessing potential harms from large language models. Our contributions are first, empirical evidence of practitioners conceptualizing red teaming and developing and evaluating red teaming datasets. Second, we reflect on how practitioners’ conceptualization of risk leads to overlooking the context, interaction type, and user specificity. We conclude with three opportunities for HCI researchers to expand the conceptualization and data practices for red-teaming.

Introduction. Red teaming originates from the context of security, where adversaries simulate attacks to uncover vulnerabilities [8, 28, 74, 91]. Nowadays, red teaming has become central to evaluating the safety and robustness of systems, especially with the rise of large language models (LLMs). LLMs now outperform benchmarks in reasoning and language understanding [58], but also produce harmful, biased, or misleading outputs. These risks have made red teaming an emerging standard practice in industry, government, and academia, embedded in regulations such as the European Union’s Artificial Intelligence Act [27]. Adversarial datasets are central to red teaming LLMs, revealing models’ vulnerabilities and blind spots. These datasets are collections of prompts or conversations designed to elicit harmful or unsafe behaviors, enabling systematic evaluation and alignment improvements. They define what counts as harm, determine how models are tested, and shape what risks present to downstream users. Yet datasets are not neutral: they embody the assumptions and values of those who create them [51, 65, 67]. Prior work in Human-Computer Interaction (HCI) and critical data studies has shown that annotation practices, labeling schemes, and benchmark structures not only shape technical outcomes but also social and cultural norms [51, 65, 67]. Data workers develop intuitive understandings of their data through multiple forms of intervention, including treating data as given, captured, curated, designed, or created [53]. However, little is known about how AI practitioners create, reuse, and evaluate red teaming datasets for LLMs. Different from many prior data work that analyze data produced in real-world contexts, such as social media content [7] or image recognition [51], red-teaming datasets are created with an explicitly adversarial, experimental, and testing-oriented intention. Rather than capturing naturally occurring human activity, practitioners design prompts to provoke failures, explore attack strategies, and probe the boundaries of model behavior [44]. This adversarial stance reframes data work in this context: while red-teaming data creation is highly open-ended and unstructured, it still depends on human judgment about what to include and how to validate it. Most red teaming LLMs research emphasizes technical benchmarks and attack success rates, leaving the socio-technical practices of dataset development under-examined. These refer to the intertwined social and technical choices about what should be supported socially, and what can be implemented technically [2], such as how harms are defined, how adversarial prompts or dialogue are created, and what evaluation metrics are adopted. This gap matters because harmfulness is not discovered as a fixed concept in red teaming evaluation, but constructed through choices about prompts, categories, evaluation metrics, and collaborations. Decisions about what harms to include or exclude, and how to measure success, shape not only the coverage of datasets but also the guardrails and mitigations that follow. Understanding these practices is critical for HCI, which has long examined how technologies embed human values, and for AI safety, which increasingly depends on socio-technical insights. To address this gap, we conducted 22 semi-structured interviews with AI practitioners who design, build, or reuse datasets for red teaming LLMs. Our study is guided by the following research questions:

(1) How do AI practitioners create, develop, and evaluate red teaming datasets, and why in these ways? (2) What tools and support do AI practitioners need when developing red teaming datasets?

Our work extends prior HCI examinations of data work by revealing a distinct form of adversarial, interaction-driven data work that emerges uniquely in red teaming LLMs. Our analysis highlights three critical moments in this work: (1) defining and framing red teaming tasks, (2) developing adversarial datasets, and (3) evaluating models against them. Across these moments, we show that disciplinary backgrounds shape whether red teaming is framed as exploration or classification, while motivations range from observing “in-the-wild” jailbreaks to addressing technical or regulatory gaps. We find that datasets are developed through different pathways: created from scratch, repurposed from existing resources, or derived from human interactions, and that evaluation is complicated by issues of context, diversity, and metrics. Taken together, these practices reveal red teaming as more interactional and social than AI practitioners typically anticipated in benchmark-driven work, with risks arising not only from single prompts but also from multi-turn, multilingual, and multicultural exchanges.

Related work. To contextualize our contribution, we first introduce the concept and practice of red teaming, then examine how recent datasets operationalize adversarial testing for LLMs. We then consider the diverse stakeholders involved in red teaming for AI safety, highlighting the motivations and tensions that shape its practice. Finally, we situate our work within HCI research, showing how longstanding contributions on data practices provide critical insights for understanding and improving red teaming evaluation.

Red teaming did not originate as a technical protocol, but rather as a strategic exercise in adversarial thinking. Rooted in military war-gaming, red teams were tasked with challenging institutional assumptions by “thinking like the enemy” to expose vulnerabilities in operational plans [8, 74, 91]. Over time, this practice migrated into cybersecurity, where red teams simulate breaches to identify flaws in digital defenses, a tradition now formalized in frameworks like the U.S. Department of Defense’s Cyber Red Teaming Guide [28, 74]. This adversarial framing of treating safety as a matter of defense against active threats shaped how risk was conceptualized in digital systems, establishing a dominant view of harm as something imposed by hostile actors, rather than as the emergent, everyday vulnerabilities experienced by diverse users [3, 28]. It provided a blueprint for how harms are anticipated, operationalized, and mitigated in sociotechnical design [32, 75]. Academic scholarship across security studies [47], information systems [10], CSCW and HCI [92–94] contributed to this transition, theorizing red teaming as a method of system auditing, threat modeling, and scenario-based testing. These foundational contributions laid the groundwork for the widespread adoption of red teaming in industry, government, and research settings. In the domain of AI, and particularly with the rise of LLMs and emerging agentic AI systems, red teaming has taken on renewed urgency [3, 28, 84]. While LLMs offer substantial societal benefits, including expanded access to information, enhanced productivity, and support for human creative and cognitive tasks [33, 79, 95], they have also been shown to generate outputs that are toxic, biased, or sometimes inaccurate [83]. In response, the Center for AI Safety released a statement in 2023, warning that AI extinction risk should be a global priority [19]. Therefore, governments (e.g., the EU’s AI Act [27] and the U.S. Executive Order 14110 [36]) and industry leaders (e.g., OpenAI [57], Anthropic [9]) have institutionalized red teaming as a key part of AI evaluation.

Method. In this section, we provide an overview of our interview study. We describe the recruitment strategy we follow to ensure that we interview practitioners with red teaming expertise, and provide a characterization of the participants’ profiles. We conclude with a description of our data analysis process.

To understand how red teaming datasets are designed, developed, and evaluated, we recruit AI practitioners who have released public red teaming datasets or have experience researching red teaming datasets. Our process consisted of first identifying potential participants with experience in red teaming datasets for LLMs. We searched for red teaming datasets from Hugging Face 1 and Papers with Code platforms 2. We searched ‘red teaming’ in full-text search in Hugging Face to collect red teaming datasets and the data creators’ information. We stopped after the top 100 search results. Because in the latter portion of the 100, there are many duplicated datasets from higher-ranked original red teaming datasets. Then we searched ‘red teaming’ on Papers with Code, which gave us 100 results of papers and their GitHub repositories. We also explored additional communities, such as Kaggle, OpenML, and the Data Provenance Initiative, but only found five red teaming datasets3 We conducted 22 semi-structured interviews with AI practitioners, and each interview took between 40 to 60 minutes. Nineteen interviews were conducted in English, two were in Chinese, and one was in Spanish. The study was approved by the authors’ Institutional Review Board. Participation was voluntary. We informed the participants that all information they shared would be kept confidential and anonymous, and they permitted us to record the interview for further analysis. We chose semi-structured interviews as our primary method because they offer an appropriate way to understand practitioners’ rationales when developing red teaming techniques and provide critical access to settings where ethnographic entry is constrained. Opportunities for fieldwork or ethnographic studies are limited across both industry and academia: red-teaming activities in industry often occur within proprietary or confidential environments [12], while academic red-teaming projects are also typically confidential until publication. Therefore, we consider interviewing as an appropriate mode of inquiry to understand practitioners’ day-to-day activities that otherwise would remain inaccessible. Interviews also enabled us to reach participants working across diverse geographic and institutional contexts, which would have been logistically challenging to access through site-based fieldwork. While we recognize that observational methods could help uncover the tacit and embodied aspects of data work, they limit the ability to surface the rationales, trade-offs, constraints, and assumptions that guide people’s choices. In contrast, semi-structured interviews allow for eliciting accounts of not only what people do but why they do it. This is particularly important for examining the assumptions embedded in dataset construction and evaluation practices, which are often invisible in observational inquiry alone. To reduce idealized recall, we grounded our interview protocols in participants’ publicly released datasets and articles focused on red teaming techniques. We asked about the challenges and limitations they experienced, and asked them to explain their practice decisions in their work. Before each interview, we reviewed the participants’ published papers and Hugging Face dataset cards, if applicable, to familiarize ourselves with their work and prepare our interview questions. We started the interviews by learning the participants’ experiences in red teaming. Then we went deeper to discuss participants’ experiences in creating or reusing red teaming datasets, their motivation for red teaming research, their categorization of harm and risks, and evaluation approaches. For each interview, we tailored, added, or skipped some questions towards the participant’s own experiences and insights.

Discussion. While red teaming is often framed as a technical practice aimed at detecting model vulnerabilities, studies have shown that the definition of “risk” is not universal but reflects the values, priorities, and threat models of those in charge of defining what is tested and how [28, 32, 72]. We observed this dynamic in practice. Our findings are organized into three sections, illustrating how practitioners define and operationalize risks across three moments in red teaming LLMs: (1) conceptualizing what red teaming means, (2) developing adversarial datasets to capture those risks, and (3) evaluating the outcomes of red teaming LLMs.

In this section, we illustrated how practitioners conceptualize red teaming LLMs by examining their definitions of red teaming, their motivations for red teaming LLMs, and their views of red teaming as an interactional and social process.

4.1.1 Participants’ definitions of red teaming LLMs. Most participants started working on red teaming LLMs around the time LLMs became widely accessible and began outperforming standard benchmarks in commonsense reasoning and language understanding [58], raising concerns about safety and misuse. Six participants had worked on red teaming for about two years, seven had one