Moral Crumple Zones: Cautionary Tales in Human-Robot Interaction

Source: Madeleine Clare Elish; Engaging Science, Technology, and Society 5, journal article (open access) · 2019

MADELEINE CLARE ELISH1 DATA & SOCIETY RESEARCH INSTITUTE Abstract As debates about the policy and ethical implications of AI systems grow, it will be increasingly important to accurately locate who is responsible when agency is distributed in a system and control over an action is mediated through time and space. Analyzing several high-profile accidents involving complex and automated socio-technical systems and the media coverage that surrounded them, I introduce the concept of a moral crumple zone to describe how responsibility for an action may be misattributed to a human actor who had limited control over the behavior of an automated or autonomous system. Just as the crumple zone in a car is designed to absorb the force of impact in a crash, the human in a highly complex and automated system may become simply a component—accidentally or intentionally—that bears the brunt of the moral and legal responsibilities when the overall system malfunctions. While the crumple zone in a car is meant to protect the human driver, the moral crumple zone protects the integrity of the technological system, at the expense of the nearest human operator. The concept is both a challenge to and an opportunity for the design and regulation of human-robot systems. At stake in articulating moral crumple zones is not only the misattribution of responsibility but also the ways in which new forms of consumer and worker harm may develop in new complex, automated, or purported autonomous technologies. Keywords autonomous vehicles; responsibility; machine learning; human factors; accidents; social perceptions of technology; self-driving cars, robot; human-in-the-loop; human-robot interaction Introduction On March 18, 2018, a self-driving Uber car struck and killed a pedestrian crossing her bike in the middle of an Arizona roadway. At the steering wheel of the putative “autonomous vehicle,” a safety driver sat. Her job was to monitor the car’s systems and take over in the event of an emergency. The safety driver now may face criminal charges of vehicular manslaughter (Somerville and Shepardon 2018). A devastating accident has forced the question that had been

Questions having to do with the responsibility of various agents in complex, computational systems are not new. Such issues have been looked at from the diverse perspectives of law and policy (Calo, Froomkin and Kerr 2016) human factors engineering (Cummings 2006; Sheridan and Parasuraman 2005), systems design (Friedman 1997; Leveson 2011), ethics (Lin, Abney, and Bekey 2014; Coeckelbergh 2011; Bryson, Diamantis and Grant 2017) and the sociology of risk and innovation (Perrow 1984; Vaughn 1996). This article aims to be in conversation with these literatures and to contribute to ongoing public debates about AI and autonomous systems by providing a concept with which to think about the construction and attribution of responsibility in complex “intelligent” systems. Specifically, I articulate the concept of a moral crumple zone to describe how responsibility for an action may be misattributed to a human actor who had limited control over the behavior of an automated or autonomous system.

2 Just as the crumple zone in a car is designed to absorb the force of impact in a crash, the human in a highly complex and automated system may become simply a component—accidentally or intentionally—that bears the brunt of the moral and legal responsibilities when the overall system malfunctions. While the crumple zone in a car is meant to protect the human driver, the moral crumple zone protects the integrity of the technological system, at the expense of the nearest human operator. What is unique about the concept of a moral crumple zone is that it highlights how structural features of a system and the media’s portrayal of accidents may inadvertently take advantage of human operators (and their tendency to become “liability sponges”) to fill the gaps in accountability that may arise in the context of new and complex systems.

Building on previous work analyzing the history of aviation autopilot litigation in the 20 3 this article calls attention to how incongruities between control and responsibility arise in complex systems, and how, in turn, such mismatches shape public and expert perceptions of responsibility. First, I present two well-known accidents involving complex socio-technical systems in which I argue moral crumple zones emerge, the partial nuclear meltdown at Three Mile Island and the crash of Air France Flight 447. My aim is not to provide new insight into the causes or circumstances surrounding the accidents. Rather, my aim is to call attention to a dynamic within complex systems that I argue will be increasingly relevant in the Madeleine Clare Elish Engaging Science, Technology, and Society 5 (2019) context of autonomous, robotic, and AI systems. The circumstances surrounding and the immediate responses to and media coverage of these accidents demonstrate how accountability appears to be deflected off of the automated parts of the system (and the humans whose control is mediated through this automation) and focused on the immediate human operators, who possess only limited knowledge, capacity, or control.

Following the discussion of these cases, I apply the concept of the “moral crumple zone,” in which human operators take on the blame for errors or accidents not entirely in their control, to accidents involving self-driving cars. In this context, I call attention to the ways in which users and operators of such systems may be held responsible for failures in ways that obscure other human actors who may possess equal if not greater control over the behavior of a purportedly “autonomous” system. At stake in articulating “moral crumple zones” is not only the misattribution of responsibility but also the ways in which new forms of consumer and worker harm may develop in new automated technologies. The article concludes by discussing the implications of these dynamics for future policy and regulatory decisions.

Method. This counterexample brings into relief the characteristics of systems or accidents in which we might expect to see moral crumple zones emerge. First, moral crumple zones are likely to take shape in the immediate aftermath of a highly publicized event or accident. It is notable that the Therac-25 accidents became public nearly two years after the first overdose of radiation was given. The malfunction and harm was not immediately clear, but rather came to light slowly. In other words, when the accidents were reported, months of inquiries into the causes of the error had taken place, and fault had already been assigned to the manufacturer. Moreover, media coverage plays a significant role in shaping social perceptions of responsibility and accountability and in creating the context in which moral crumple zones emerge and take hold.

Second, we can see a difference in the position in which the operator of the system was placed. In the case of Therac-25, the operator had no way of knowing that the system had malfunctioned, except for reports from patients that felt pain. In the case of the TMI meltdown, the operators knew the system was malfunctioning, but they did not have sufficient information or authority to take corrective actions. In the case of the Air France crash, the extent to which the system was malfunctioning was only variously visible, and while the pilots had sufficient information to react, in many ways they were systemically disempowered to act appropriately on that information Moreover, the Therac-25 failed to perform as designed, and this failure to carry out the intended action resulted in a radiation overdose. In the case of TMI, the system performed as designed––but ultimately it was a failure of how the system, itself, was designed and maintained that created the conditions for a partial meltdown to take place. In the case of the Air France crash, once again, the system performed as designed in the face of a mechanical failure and still the accident occurred. This dynamic underscores how a moral crumple zone is more than just the articulation of a scapegoat. The term is meant to call attention to the ways in which automated and autonomous systems deflect responsibility in unique and structural ways, Madeleine Clare Elish Engaging Science, Technology, and Society 5 (2019) protecting the integrity of the technological system at the expense of the nearest human operator. The technology is maintained as faultless, while the human operator becomes the faulty feature of the system.

Discussion. Without a doubt, actions taken by the plant operators led to the accident and exacerbated its severity. A maintenance worker two days prior had indeed left a valve closed after a testing procedure that should have been left open. It was steps taken by a maintenance worker to fix a clogged pipe that resulted in halting circulation in the feedwater pipes. And it was operators in the control room who overrode the final safety system, which would have engaged the remaining backup water system. But to focus on these actions as isolated events is like focusing on a detail in the foreground while missing the bigger picture.

For instance, the design of the control room played a central role in compounding human misinterpretations of mechanical failures. Designed as an automated system with limited human oversight, the physical conditions of the system were not adequately represented in the control interface (Rubinstein 1979; Sheridan 1992). For instance, there were no direct indicators of the level of cooling water in the steam generator tank. The automated system received this information (which had triggered the automatic shutdown), but the operators had to infer the amount of water from an auxiliary tank linked to pressure monitoring. During the accident, this tank remained full and provided incorrect information about the system to the operators. The operators made incorrect decisions because they had incorrect information. One of the central recommendations of the report was the requirement to focus on human factors engineering and the importance of human-computer interaction design (Kemeny et al. 1979).

Additionally, if the frame is expanded beyond those immediately present during the accident, errors followed directly from other systemic errors. It later came to light that the workers had been directed to test the valves and document the testing in a way that cut corners Focusing only on the agency of operators misses other sites of interaction and dimensions of control exercised by other actors involved in the system, from the designers of the interfaces to the plant managers who created the conditions within which the operators could act, to the regulators who maintained a blind-eye toward industry standards. It was this level of system complexity in which interactions were tightly coupled that Perrow (1984) points out as necessarily producing system failure in the form of a “normal accident” at TMI. His and later accounts (Walker 2004) emphasize the systemic causes that contributed to the accident, and underscore the incompleteness of understanding the accident as the result of human error.

Still, news coverage and later popular accounts of the accident positioned the operators as the moral crumple zone of the system. Even while the Kemeny report (1979) highlighted the reigning “mindset” at the plant and how “systemic” problems were the basis for the accident, the report emphasized the role of human failures in contrast to functioning technology, stating in the early pages of the report, The partial nuclear meltdown at TMI is a useful example through which to see how a mismatch might emerge between the actual and the imagined or perceived control over a system that an operator may have. Even as their actions were constrained, their culpability was focused on and singled out. They became the moral crumple zone for the system’s failure in the eyes of many. The Crash of Air France Flight 447 Although the partial meltdown at Three Mile Island occurred several decades ago, the challenges of shared control between humans and machines that contributed to the accident remain essentially unsolved.

Conclusion. This article has attempted to articulate a problem and characterize a set of frictions that emerge when automated systems disrupt traditional linkages between control and responsibility. The discussion has ultimately been two-fold. In the first part, I articulated the potential mismatches that can occur between control and responsibility in automated systems through a discussion of the nuclear meltdown at Three Mile Island and the crash of Air France Flight 447. These mismatches, I argued, create moral crumple zones, in which human operators take on the blame for errors or accidents not entirely in their control. In the final part of the article, I brought the idea of the moral crumple zone out of the context of industrial systems and asked what it might look like in the context of commercial technologies, namely, driverless cars. I also explored how traditional modes of technology certification may reify the potential to create moral crumple zones and suggested that a reexamination of certification paradigms may be a productive avenue of future research.

This article presents the concept of the “moral crumple zone” as both a challenge to and an opportunity for the design and regulation of human-robot systems. At stake in the concept of the moral crumple zone is not only how accountability may be distributed in any robotic or autonomous system, but also how the value and potential of humans may be allowed to develop in the context of human-machine teams.

Madeleine Clare Elish Engaging Science, Technology, and Society 5 (2019)