Can workplace AI risks emerge from interactions alone?
This explores whether AI agent risks can arise from how agents, goals, environments and humans interact together, even when each component functions correctly. The question matters because it suggests safety requires systems-level thinking, not just component-by-component testing.
The paper's framework, built from a literature review on AI agents, "models risk as emerging from the properties of three core components (agents, goals, environments) and from the interactions between them and humans." Together with the humans they work alongside, these form what the authors call an agentic AI system. The discussion states why the decomposition matters: "interaction-driven risks can arise even when no single component is malfunctioning, a pattern our empirical findings confirm." The alternative it rejects is treating an AI system "as a black box."
The reasoning has two parts. First, the risk pathways are wider than agent-to-agent communication. The paper places itself as extending Hammond et al. 2025, which focused on agent–agent communication, by showing that "goal and environment mediation are equally important risk pathways." Second, human interaction is not one category. The framework splits it into five relationship types, of which the excerpt names agent–human, human–human and human–environment before an "etc." The motivation given in the introduction is practical: organizations red-team agents, run impact assessments and audit incidents, and existing taxonomies "focus on broad risks and do not capture the job-specific risks introduced by agents." The framework was then embedded in a structured prompt, applied to 2,078 O*NET tasks, and the resulting 8,356 scenarios were used to extend an existing taxonomy into 15 categories.
Against the neighbors, the closest match is How do agent security layers connect across the stack?, which reaches a similar conclusion from the security side: layers have to be considered together. The workplace paper slices the system along different lines, into agents, goals, environments and humans, rather than along stack layers. Who actually bears the risk when multi-agent workflows fail? separates requester, observer and affected party, and that split looks like a case of the human–human relationship types this framework counts separately. The excerpt itself does not discuss third parties. Does a single benchmark score actually predict agent readiness? argues for decomposing capability, and this paper makes the risk-side companion point: passing checks on each component would not show that the interactions are safe. That companion reading is the vault's, not the paper's. The sibling note Does AI augmentation protect workers from skill erosion? reads as one instance, since overreliance concerns the worker–agent relationship and does not require the agent to fail.
What the excerpt does not establish. It gives no count or example of interaction-driven scenarios, so the claim that the findings "confirm" the pattern cannot be checked here. It names only three of the five human relationship types and none of the 15 taxonomy categories. The taxonomy was extended to cover all of the scenarios, and those scenarios were generated with the framework, so coverage alone does not show the decomposition beats an undecomposed taxonomy. The excerpt reports no such comparison. What follows at this strength is a design stance for risk assessment: list the goals, environment and human relationships alongside the agent, because the excerpt's claim is that a component-by-component review can come back clean while a risk remains.
Inquiring lines that read this note 7
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
How can humans maintain meaningful oversight as AI systems become increasingly autonomous and complex?- How does autonomy level shape the kinds of risks AI agents pose?
- Does low autonomy AI inherently create different risks than high autonomy AI?
Related concepts in this collection 5
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
-
How do agent security layers connect across the stack?
Agent security is often treated as separate challenges at each layer—inputs, delegation, routing, containment. But do defenses at one layer fail if others aren't secured? This explores whether securing agents requires end-to-end integration.
parallel conclusion from security: risk sits across layers, here across agents, goals, environments and humans
-
Who actually bears the risk when multi-agent workflows fail?
When AI agents delegate tasks across organizations, the people harmed by failures may never see the workflow or author the prompts. This explores whether current oversight designs protect the right parties.
the requester, observer and affected split resembles the human–human relationship types counted separately here
-
Does a single benchmark score actually predict agent readiness?
Single-axis benchmarks rank models by one capability—like task success—but ignore privacy, duration, operating mode, and ecosystem fit. Can one number really capture what matters for deployment?
decomposition on the capability side; this note is the risk-side counterpart
-
Does AI augmentation protect workers from skill erosion?
Workplace AI labeled as augmentation is often considered safer than automation because humans stay involved. But does relying on AI agents to assist work actually preserve or gradually erode worker skills and their ability to oversee the system?
sibling: a risk of the worker–agent interaction that needs no malfunctioning agent
-
Can individually safe agents fail when working together?
When multiple AI agents interact—sharing information, state, and authority—do failures emerge that local safety checks alone cannot catch? This matters because system-level safety depends on understanding how principals interact.
Extends to multi-agent systems: benign fragments, truthful reports and honest specialists, each harmless alone, produce harm once information, state and authority cross principal boundaries
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- Unaccountable Delegation, Fading Skills: Mapping the Risks of Workplace AI Agents
- Utility Engineering: Analyzing and Controlling Emergent Value Systems in AIs
- Psychological Influences of Conversational AI: Research and Design Directions for Reducing Harm and Promoting Well-Being
- Fully Autonomous AI Agents Should Not be Developed
- Agentic Misalignment: How LLMs Could Be Insider Threats
- Frontier AI Risk Management Framework in Practice: A Risk Analysis Technical Report
- The safety failures we are not instrumenting: a perspective on hidden safety-critical challenges in modern AI systems
- Agents of Chaos
Original note title
interaction-driven risks of workplace AI agents can arise even when no single component is malfunctioning — goals and environments mediate risk too