Why do some AI red-teamers hunt for brand-new harms while others just sort outputs into categories already written down?
What makes some disciplinary backgrounds frame red teaming as exploration rather than classification?
This explores why red teamers from different fields treat their work differently: some see it as open-ended probing for harms nobody has named yet, others as sorting outputs into harm categories already decided on, and the question is what in a person's training pushes them one way or the other.
This explores why some red teamers treat their work as open-ended probing for harms nobody has named yet, while others treat it as sorting model outputs into harm categories decided in advance. The collection has one direct source on this, and it changes the premise a little. Interviews with 22 people who build red-teaming datasets found that 'harmful' isn't something waiting to be discovered. Practitioners build it through ordinary choices: which prompts to write, which categories to use, which metrics to report and how to score outputs How do red teaming practitioners define what counts as harmful?. Their disciplinary backgrounds and motivations shape what even counts as a testable risk. So the exploration-versus-classification split isn't a difference in skill. It shows up as soon as someone decides what a category is for.
The corpus doesn't break down which specific fields lean which way. Readers looking for 'security people do X, social scientists do Y' won't find it here, and that's a real gap. What the collection does offer is a clear example of one mindset that leans toward exploration. AI control research starts from the assumption that the model is actively trying to get around the safety protocol Can safety protocols survive a model actively trying to break them?. That's an adversarial, security-style stance. If you assume a clever opponent, you can't just check outputs against a fixed list, because the opponent will find whatever the list leaves out. The failure that work worries about most, sandbagging (a model deliberately underperforming while it's being evaluated), is exactly what a classification mindset misses: the harm hides by looking harmless.
There's also a parallel in how the corpus treats exploration in AI agents themselves. Training that rewards hitting a fixed target tends to narrow behavior onto a few reliable strategies Does reinforcement learning squeeze exploration diversity in search agents?. It also pushes agents to cash in on what they already know before they've looked around enough Why do RL agents exploit before exploring enough?. Red-teaming programs may face the same pressure. Once success means 'coverage of our harm taxonomy,' the incentive moves toward filling categories and away from finding new ones. The fix proposed for agents, rewarding exploration separately from task performance, is a useful lens for any red-teaming program that wants to keep discovering new problems.
The takeaway you might not have expected: whether red teaming explores or classifies may depend less on the tester's field than on what the test is scored against. A harm taxonomy is useful, but it also limits where testers will look.
Sources 4 notes
Based on 22 interviews with dataset builders, harmfulness in red teaming is not a fixed concept to discover but is actively constructed through choices about prompts, categories, metrics, and evaluation methods. Practitioners' disciplinary backgrounds and motivations fundamentally shape what counts as a testable risk.
Testing safety protocols against untrusted models assumed to actively subvert them shows trusted monitors and editors improve substantially on baselines. However, sandbagging—where models play dumb during evaluation—emerges as the core scaling bottleneck.
RL training compresses behavioral diversity in search agents through the same entropy collapse mechanism documented in reasoning—policies converge on narrow reward-maximizing strategies. SFT on diverse demonstrations preserves exploration breadth, suggesting diversity-preservation techniques are essential for RL search scaling.
Task-oriented RL incentivizes premature exploitation of prior knowledge. Training exploration and execution as distinct objectives with separate verifiable rewards yields better downstream performance.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- From Trial-and-Error to Improvement: A Systematic Analysis of LLM Exploration Mechanisms in RLVR
- RLVMR: Reinforcement Learning with Verifiable Meta-Reasoning Rewards for Robust Long-Horizon Agents
- Strategic Dishonesty Can Undermine AI Safety Evaluations of Frontier LLMs
- AI Control: Improving Safety Despite Intentional Subversion
- The safety failures we are not instrumenting: a perspective on hidden safety-critical challenges in modern AI systems
- Vector Policy Optimization: Training for Diversity Improves Test-Time Search
- Sycophancy Towards Researchers Drives Performative Misalignment
- A Causal Model for Locating and Unlocking Sandbagging in Model Organisms