How do red teaming practitioners define what counts as harmful?

Based on 22 semi-structured interviews with AI practitioners who design, build, or reuse red-teaming datasets for large language models, the paper's authors argue that red teaming is a socio-technical practice rather than a purely technical exercise. The abstract states that existing work emphasizes "technical benchmarks and attack success rates, leaving the socio-technical practices of how red teaming datasets are defined, created, and evaluated under-examined." Their interviews, with practitioners recruited by searching "red teaming" on Hugging Face and Papers with Code, found that "harmfulness is not discovered as a fixed concept in red teaming evaluation, but constructed through choices about prompts, categories, evaluation metrics, and collaborations."

The authors organize practitioners' work into three moments: "(1) defining and framing red teaming tasks, (2) developing adversarial datasets, and (3) evaluating models against them." Across these moments, they find datasets are built through distinct pathways — "created from scratch, repurposed from existing resources, or derived from human interactions" — and that "disciplinary backgrounds shape whether red teaming is framed as exploration or classification," with motivations ranging from observing "in-the-wild jailbreaks" to addressing technical or regulatory gaps. Evaluation, in turn, is "complicated by issues of context, diversity, and metrics," with risks arising not only from single prompts but from "multi-turn, multilingual, and multicultural exchanges."

This reframes benchmark-driven safety evaluation as a human data-construction problem rather than a model-measurement problem. It parallels Does a single benchmark score actually predict agent readiness?, which argues capability itself is multi-axis and under-measured by any single benchmark — here the missing axis is the practitioner's own disciplinary framing of what counts as harm. It contrasts with Where do frontier AI models actually pose the greatest risk today?, which treats risk as a set of fixed, measurable capability thresholds, where this paper finds the categories themselves are practitioner-constructed rather than given. It also connects to Is evaluation awareness really one unified capability?: both question whether red-teaming-style evaluation measures a stable target, one locating the instability in model behavior under framing, the other in the human choices that build the adversarial dataset itself. And it contrasts with Can safety protocols survive a model actively trying to break them?, whose adversarial stance assumes an actively subverting model, where this paper's adversarial stance is practitioners designing prompts to provoke failures — a human-centered rather than model-centered notion of "red team."

The excerpt is a partial draft, cutting off mid-sentence inside the first results subsection on participants' definitions of red teaming, so the full empirical breakdown of how many practitioners used each dataset pathway, or the paper's "three opportunities for HCI researchers," is not available here. The sample is also self-selected: participants were identified by searching "red teaming" on Hugging Face and Papers with Code and stopping after the top 100 results, which the authors note surfaced many duplicated datasets, and the 22 interviews skew toward English speakers (nineteen in English, two in Chinese, one in Spanish). The excerpt does not establish how widespread any single disciplinary framing is, only that it varies across practitioners; the implication is that red-teaming benchmarks and leaderboards inherit the risk definitions of whoever built the dataset, which is a reason to treat red-teaming scores as contingent on practitioner judgment rather than as neutral measurements of model safety.