Before switching on a high-risk AI system, who has to work out exactly how it could hurt the people it touches?

This explores what a deployer of a high-risk AI system has to assess about its effects on people's rights before going live, and what makes such an assessment more than a box-ticking exercise. First, a direct caveat: the corpus has no notes on the fundamental rights impact assessment itself. For reference, and not drawn from the corpus: under Article 27 of the EU AI Act, deployers such as public bodies and some credit and insurance providers must describe how the system will be used, how often and for how long, who will be affected, what specific harms those people face, what human oversight is in place, and what happens if the risks materialize. They must then notify the regulator. What the corpus does offer is a sharp set of reasons why each of those requirements is harder to meet than it sounds.

Start with "who is affected and what could harm them." The corpus suggests that many harms don't show up in any single output. They build up over a relationship. Research on agentic conversational AI argues that the standard helpful-honest-harmless checks miss harms that emerge across repeated interactions, such as a gradual loss of a user's sense of autonomy or competence Can HHH criteria alone capture harms from agentic AI?. An assessment that only audits individual decisions could pass a system that slowly wears away the very rights it is meant to protect. A related blind spot is that the people most affected often can't see the rules governing the system. System prompts form a confidential hierarchy that developers control and end users never see, and small edits can abruptly change behavior Who actually controls what AI systems say?. A one-time assessment can't capture a system that keeps changing after it is approved.

Next, "human oversight measures." One finding gives a useful rule of thumb: risk to people rises steadily with the autonomy you hand to an AI agent, so the oversight you need depends on where on the autonomy spectrum you deploy Does AI risk increase with the autonomy we give it?. A design-side argument adds that, for autonomous systems, the critical safeguard is specifying limits, forbidden actions and risk tolerance before the system acts. A vague instruction carries through every later action, and no human reviews those actions Should agentic AI interfaces require users to specify risk tolerance upfront?. Read this way, an impact assessment is partly an intent specification written for affected third parties rather than for the operator.

The most surprising thread concerns "what happens if the risks materialize." A paper asking how you would actually measure whether a system's errors stay visible, contained and recoverable concludes that the tools don't yet exist as a set. There are partial measures for each part, but nothing captures the whole system, including the humans and institutions around the model How can we measure whether AI errors stay visible and recoverable?. That is close to what a rights assessment promises to deliver. Meanwhile, a long-running case study of a persistent agent found that safeguards worked best when they were written into the memory the agent consulted while making decisions, not kept as an outside policy document Can governance rules embedded in runtime memory actually protect autonomous agents?. If that holds more broadly, an impact assessment filed with a regulator is the weakest form of protection. The stronger form is one whose commitments live inside the running system.

Finally, the reason assessments are required by law rather than left voluntary appears in the corpus too: the argument that companies cannot reliably police AI risk on their own without binding outside oversight Can companies alone manage the risks of AI systems?. What you may not have expected is the gap these notes point to together. The law asks deployers to show that harms will be caught and corrected. The research suggests that harms often build up gradually, sit in layers users can't see, and fall through measurement tools that don't yet cover the whole system.