Human-Centered Artificial Intelligence: Reliable, Safe & Trustworthy

Well-designed technologies that offer high levels of human control and high levels of computer automation can increase human performance, leading to wider adoption. The Human-Centered Artificial Intelligence (HCAI) framework clarifies how to (1) design for high levels of human control and high levels of computer automation so as to increase human performance, (2) understand the situations in which full human control or full computer control are necessary, and (3) avoid the dangers of excessive human control or excessive computer control. The methods of HCAI are more likely to produce designs that are Reliable, Safe & Trustworthy (RST). Achieving these goals will dramatically increase human performance, while supporting human self-efficacy, mastery, creativity, and responsibility.

Introduction. This paper opens up new possibilities by way of a two-dimensional framework of Human- Centered Artificial Intelligence (HCAI) that separates levels of automation/autonomy from levels of human control. The new goal is to seek high levels of human control AND high levels of automation, which is more likely to produce computer applications that are Reliable, Safe & Trustworthy (RST). Achieving this goal, especially for complex poorly understood problems, will dramatically increase human performance, while supporting human self-efficacy, mastery, creativity, and responsibility.

The traditional belief in computer autonomy is compelling for many artificial intelligence (AI) researchers, developers, journalists, and promoters. The goal of computer autonomy was central in Sheridan and Verplank’s (1978) ten levels from human control to computer automation/autonomy (Table 1). Their widely cited one-dimensional list continues to guide much of the research and development, suggesting that increases in automation must come at the cost of lowering human control. Shifting to HCAI could liberate design thinking so as to produce computer applications that increase automation, while amplifying, augmenting, enhancing, and empowering people to innovatively apply systems and creatively refine them.

Critics of autonomy have repeatedly discussed the ironies (Bainbridge, 1983), deadly myths (Bradshaw et al., 2013; Mindell, 2015), conundrums (Endsley, 2017), or paradoxes (Hancock, 2017) of autonomy. A common point is that humans have to spend more effort monitoring autonomous computers because they are unsure of what it will do, often leading to inferior performance (Blackhurst et al., 2011; Strauch, 2017).

Fortunately, there is growing awareness by leading artificial intelligence researchers and developers that human-centered designs are needed (Jordan, 2018; Li, 2018). Equally important is the encouragement of creativity researchers to integrate creativity support features in technology (Candy, 2020; Edmonds, 2020; Edmonds and Candy, 2002).

Related work. These stages refine discussions of each of the levels, but the underlying message is that the goal is full automation/autonomy.

Even Sheridan (2000) commented with concern that “surprisingly, the level descriptions as published have been taken more seriously than were expected” (see Hoffman & Johnson (2019) for a detailed history). However, in spite of the many critiques, the 1-dimensional levels of automation/autonomy, which only represents situations where increased automation must come with less human control, is still widely influential. For example, the US Society of Automotive Engineers adopted the unnecessary trade-off in its six levels of autonomy for self-driving cars (SAE, 2014; Brooks, 2017) (Table 2).

Bradshaw et al. (2013) caution designers who “have succumbed to myths of autonomy that are not only damaging in their own right but are also damaging by their continued propagation ... because they engender a host of other serious misconceptions and consequences.” The U.S. Defense Science Board (Murphy and Shields, 2012) report describes the ten levels of automation/autonomy this way: “though attractive, the conceptualization of levels of autonomy as a scientific grounding for a developmental roadmap has been unproductive.” A later report described many opportunities and dangers (Defense Science Board 2016), while cognitive science researchers (Hoffman et al., 2016) point to the failures and costs of autonomous weapons.

Method. My discussion of Reliable, Safe & Trustworthy clarifies what leads to high performance. These definitions emphasize (1) technical practices that support reliability, (2) management strategies that create cultures of safety, and (3) independent oversight structures that support trust.

Reliable (Modarres et al., 2016) systems come from appropriate technical practices, which support human responsibility (Canadian Government, 2019), fairness (O’Neil, 2016), and explainability (Du, Liu & Hu, 2019), such as:

Cultures of safety (Guldenmund, 2000; Berry et al., 2016) are cultivated by open management strategies such as There is a long history of discussions to define trust such as Fukuyama (1995), who focused on social trust “within a community of regular, honest, and cooperative behavior, based on commonly shared norms, on the part of the members of that community.” His community-based definition leads to raising trust by relying on respected independent oversight structures (Shneiderman, 2000, 2016) including:

The HCAI framework in this paper steers designers and researchers to ask fresh questions and rethink the nature of automation/autonomy. As designers get beyond thinking of computers as our teammates, collaborators, or partners, they are more likely to develop technologies that dramatically increase human performance by taking advantage of unique computer features including sophisticated algorithms, advanced sensors, information abundant displays, and powerful effectors. Clarifying human responsibility also guides designers to support human capacity to invent creative solutions in novel contexts with incomplete knowledge. The HCAI framework clarifies that the goal of design excellence is to promote human self-efficacy, mastery, and responsibility, which support the Reliable, Safe & Trustworthy systems that managers and users seek.

The one-dimensional levels of automation, grew from Sheridan and Verplank’s early paper, but was repeated in many articles and textbooks, including my own (Figure 1).

Designers thought they had to choose a point on the one-dimensional line from human control to computer automation. The implicit message was that more automation meant less user control. The decoupling of these concepts leads to a two-dimensional HCAI framework, which suggests that achieving high levels of human control and high levels of automation is possible (yellow triangle in Figure 2).

The desired goal is often, but not always, to be in the upper right quadrant. Most RST systems are on the right side. The lower right quadrant is home to relatively well-understood and predictable tasks, e.g. automobile automatic transmission or skid control on normal highways. For poorly understood and complex tasks with varying contexts of use, the upper right quadrant is needed. These tasks involve creative decisions, making them currently at the research frontier. As contexts of use are standardized (e.g. elevator shafts) these tasks can become under greater computer control with high levels of automation.

The lower right quadrant (Figure 3), with high automation and low human control, is home of computer autonomy requiring rapid action, for example, airbag deployment, anti-lock brakes, pacemakers, implantable defibrillators, or defensive weapons systems. In these applications, there is no time for human intervention or control. Because the price of failure is so high, these applications require extremely careful design, extensive testing, and monitoring during usage at scale to refine designs. Effective and proven designs can become RST systems with higher levels of automation and less human supervision.

The upper left quadrant, with high human control and low automation, is the home of human autonomy where human mastery is desired to enable competence building, free exploration, and creativity. Examples include bicycle riding, piano playing, baking, or playing with children.

Discussion. The notion of full car autonomy is productively giving way to a human-centered approach, which provides appropriate controls to improve safety, while addressing the numerous special cases of snow-covered roads, unusual vehicles, vandalism, malicious interference, construction zones, and emergency guidance from police, fire, or ambulance personnel. As car sensors, highway infrastructure, and car-to-car communication improve, strategies to make cars as safe as elevators will become easier to implement. self-driving cars by 2040 could be done with high automation and high human control, while avoiding the dangers of excessive automation or excessive human control Example 1: Patient Controlled Analgesia (PCA) devices (Figure 6) allow post-surgical, severe cancer, or hospice patients to select the amount and frequency of pain control medication. There are dangers and problems with young and old patients, but with good design and management PCA devices deliver safe and effective pain control (Macintyre, 2001). A simple morphine drip bag deaign for the lower left quadrant (low automation and low human control) would regularly deliver a fixed amount of pain control medication. A more automated design for the lower right quadrant (increased automation, but little human control) would provide machine selected doses that could vary by time of day, activity, and data from body sign sensors, although these do not assess perceived pain.

A human-centered design for the upper left quadrant (higher human control, with low automation) would allow patients to squeeze a trigger to control the dosing, frequency, and total amount of pain control medication. However, the dangers of overdosing must be controlled by an interlock that prevents frequent doses, typically with lockout periods of 6-10 minutes, and total dose limits over 1-4 hour periods. Finally, a RST design for the upper right quadrant would allow users to squeeze a trigger to get more pain medicine, but would use sensors, and machine learning to choose appropriate doses based on patient and disease variables, while preventing overdosing. Patients would be able to get information on why limiting pain medication is important with explanations of how they operate the PCA device. The RST design would include a hospital control center to monitor usage of hundreds of PCA devices so as to ensure safe practices, deal with power or other failures, review audit trails, and collect data to improve the next generation of PCA devices.

• 12 - This new goal of HCAI overcomes the 50-year debate, lucidly described by Markoff (2016), between those who sought artificial intelligence and those who sought intelligence augmentation (IA). This debate over AI vs IA now seems like arguing about iron horses vs horses or how many angels can fit on a pinhead. In fact, IBM recently declared “There’s no AI without IA”. Designers can produce HCAI by integrating artificial intelligence algorithms with user interface designs in ways that amplify, augment, enhance, and empower people.

Endsley’s “Guidelines for the design of human-autonomy systems” (2018) has 20 thoughtful items that cover human understanding of autonomous systems, minimizing complexity, and supporting situation awareness. Typical guidelines are: “Use automated assistance for carrying out routine tasks rather than higher-level cognitive functions” and “Provide automation transparency,” each of which is explained in detail. Successful designs are comprehensible, predictable, and controllable, thereby increasing the users’ self-efficacy, leading to Reliable, Safe & Trustworthy systems. These successes require careful design of the fine structure of interaction, which emerges from validated theories, clear principles, and actionable guidelines.

Conclusion. The HCAI framework separates the issue of human control from computer automation, making it clear that high levels of human control and high levels of automation can be achieved by good design. The design decisions give human operators a clear understanding of the machine state and their choices, guided by concerns such as the consequences and reversibility of mistakes. Well-designed automation, preserves human control where appropriate, thereby increasing performance and enabling creative improvements.

The HCAI framework suggests when computer control for rapid automated action is necessary, when human desire for mastery is paramount, and when there are dangers of excessive automation or excessive human control. It clarifies design choices for (1) consumer and professional applications, such as widely-used recommender systems, advertising tools, social media platforms, and search engines, which have brought strong benefits to consumers, (2) consequential applications in medical, legal, environmental, or financial systems that can bring substantial benefits and harms, and (3) life-critical applications such as cars, airplanes, trains, military systems, pacemakers, or intensive care units.

The HCAI framework is based on the belief that people are different from computers. Therefore, designs which take advantage of unique computer features including sophisticated algorithms, advanced sensors, information abundant displays, and powerful effectors are more likely to increase performance. Similarly designs and organizational structures which recognize the unique capabilities of humans will have advantages such as encouraging innovative use, supporting continuous improvement, and promoting breakthroughs to vastly improved designs.

A human-centered RST system, will be (1) reliable because of sound technical practices, (2) safe because of open management strategies that build cultures of safety, and (3) trustworthy because of well-designed independent oversight structures. RST systems will evolve rapidly because mechanisms to monitor failures and near misses support creative improvements.

Limitations. An important research direction is to develop objective measures of the levels of control and autonomy, tied to diverse tasks. Such measures would stimulate more meaningful design discussions, which would lead to improved guidelines, evaluations, and theories.

Human responsibility for mistakes is another powerful driver of design advancements such as the inclusion of detailed audit trails and consistent informative feedback about machine state. Then retraining of users and redesign of systems happens rapidly. Difficult questions remain, such as dealing with the de-skilling effects that undermine human skills that may be needed when automation fails, and the difficulty of remaining vigilant when user actions become less frequent.

Ethical questions, such as considerations of responsibility, fairness, and explainability, are helpful in developing general principles. When these general principles are combined with deep knowledge and experience with the complexities of product and service design, they can yield actionable guidelines (IEEE, 2019). The HCAI framework lays a foundation for responsibility, fairness, and explainability, but actionable guidelines to achieve these goals are beyond this paper.