Racing to the precipice: a model of artificial intelligence development

Source: AI & Society; FHI Technical Report 2013-1 (Stuart Armstrong, Nick Bostrom, Carl Shulman) · 2016 (technical report 2013)

This paper presents a simple model of an AI arms race, where several development teams race to build the first AI. Under the assumption that the first AI will be very powerful and transformative, each team is incentivised to finish first – by skimping on safety precautions if need be. This paper presents the Nash equilibrium of this process, where each team takes the correct amount of safety precautions in the arms race. Having extra development teams and extra enmity between teams can increase the danger of an AI-disaster, especially if risk taking is more important than skill in developing the AI. Surprisingly, information also increases the risks: the more teams know about each others’ capabilities (and about their own), the more the danger increases.

Introduction. This paper presents a simplified model for analysing technology races. The model was designed initially for races to construct artificial intelligences (AIs). But it can be applied to other similar races or competitions, especially technological races where there is a large advantage to reaching the goal first. There are arguments that the first true AIs are likely to be extremely powerful machines [Goo65, Cha10], but that they could end up being dangerous [Omo08, Yud08] if not carefully controlled [ASB12]. The purpose of various research projects such as ‘friendly AI’ [MS12] is to design safety precautions ensuring the creation of an AI with human-compatible values. In this paper, we won’t go that far. We will simply assume that there is a definite probability of an AI-related disaster, and that the probability goes up the more the AI development team skimps on precautions. This paper will present a model of an ‘arms race’ between AI development teams, and analyse what factors increase and decrease the probability of such an AI-disaster. Several factors contribute to increasing the danger: if building the AI depends more on risk-taking than on skill, for instance. Reducing the enmity between AI development teams helps, however, as does reducing the number of teams. But surprisingly, extra information can exacerbate the danger. The danger is minimised when each team is ignorant of the AI-building capabilities of every team – including their own capabilities. This is an interesting example of an information hazard [Bos11].

Method. In this model, there are n different teams, competing to build the first proper AI. Each team has a certain level of AI-building capability c, and can choose to take a certain level of safety precautions s, varying between s = 1 (total safety) and s = 0 (complete lack of precautions). Then each team gets a score by subtracting their safety from their capability (c −s), and the team with the highest score ‘wins’, by building their AI first. Once that AI is built, we need to check for the success of the project: That will depend on the safety level of the winning team1. With probability s, they have a successful and profitable AI project. With probability 1 −s, they have caused an AI-disaster. We’ll assume that each team maximises expected utility, and we can renormalise their utilities to give utility 1 for a successful AI project, and 0 for an AI-disaster. We will further assume that each team has the same level of enmity e towards each other team. This means that they will have a utility of 1 −e if another team builds a successful AI before they do. This varies between e = 0 (they are indifferent to who builds an AI) and e = 1 (another team building a successful AI is just as bad as an AI-disaster). It would be possible to have enmities above 1 (this would correspond to teams that hate each other so much that an AIdisaster is preferable to the triumph of the other team), but that is beyond the scope of the current paper. Further, we’ll assume that each team’s capability is drawn independently and uniformly on the interval [0, μ], for a single given μ. A high μ corresponds to a situations where capability is the dominant factor in AI development: one can achieve very little extra by skimping on precautions. Conversely, for low μ, incurring increased risk can make one much more likely to build the first AI. We’ll assume that the teams choose their safety based on Nash equilibrium considerations (this could happen in practice if, for instance, each team reduced their safety a little bit, then a little bit more, then a little bit more, in response to the other teams reducing their own safety). We’ll then calculate the Nash equilibria in three information scenarios: when teams don’t know anyone’s capabilities, when they know their own capabilities, and when they know everyone’s capabilities2.

In this situation, every team is ignorant of their own or any other team’s capabilities3. Each team will thus choose the level of safety the think is best. In In this scenario, each team knows its own capabilities, but not those of the other teams. Each team will choose a safety level s(x), dependent on their own capabilities x. In a symmetric Nash equilibrium, each team chooses the same function s. We may assume x−s(x) is a function increasing in x4. In that case, the team with the highest x, the highest capabilities, always wins. Given x, the probability of a team winning is equal to the probability of all In this scenario, every team knows the capabilities of every other team. This scenario is analysed somewhat differently that the other. Let ∆be the difference between the capability of the top team and the second ranked team. The top team always wins, but its safety level stop is determined by ∆. When stop(∆) = ∆/e, it is not in the interest of the second team to decrease its safety to compete with the top team. The gain from winning does not compensate for the extra risks run. Thus the safety of the top team will be stop = ∆/e if ∆/e < 1 and 1 otherwise. The total probability AI-disaster is calculated by integrating, across all values of ∆in [0, μ], the risk level 1−stop(∆) = 1−∆/e times the probability that the difference between the top two teams is ∆:

Discussion. Intuitively it is clear that increasing the importance of capability must decrease overall risk. One is less inclined to skimp on safety precautions if can only get a small advantage from doing so. This intuition is born out by the results: in every situation, an increase of the importance of capability (an increase in μ) reduces the risk. This is illustrated in figures 1a and 1b, as well as all subsequent figures, demonstrating that the probability of AI-disaster are always decreasing in μ. Indeed, around μ = 0 (i.e. when capability is nearly irrelevant to producing the first AI), the only Nash equilibrium is to take no safety precautions at all. In terms of intervention, there is little we can do about the relative importance of capability, since this is largely determined by the technology. Our best bet might be at present to direct research into approaches where there is little return to risk-taking, prioritising some technological paths over others.

It is also intuitively clear that reducing enmity should reduce the risk of AIdisaster. When competition gets less fierce, teams would be willing to take less risks. This intuition is also born out in our model, as decreases in e always reduce the risk. For illustration, contrast the graphs 2a and 2b, where the enmity is 0.5, with the previous graphs 1a and 1b where the enmity was 1.

Enmity is something that we can work on by, for instance, building trust between nations and groups, sharing technologies or discoveries, merging into joint projects or agreeing to common aims. With this extra coordination, we could also consider agreements to allow the teams to move away from the Nash equilibrium, thus avoiding a race to the bottom when the situation is particularly dangerous (such as low capability μ). Friendly teams make friendly AIs.

Contrasting figure 1a with figure 2a (or figure 1a with figure 2b), one notices a curious pattern. The no-information case is always safer than the other two cases, but the relative safety of private information or common information depends on the degree on enmity. It is always better if none of the teams have any idea about anyone’s capability. For maximal enmity (e = 1), the private information scenario is similarly safer than the common information one. But for lower e, this does not hold: for low μ and low e, the public information scenario can be safer than the private one. Asymptotically, though, the private information case is of order of 1/μn, while the public information case is of order 1/μ. The reasons for this is that it is only worth taking risks in the private information if one’s capability is low. So to get a high risk, one needs a winner with low capability, which is equivalent to having all teams at low capability. The probability of having a single team at low capability diminishes inversely with μ. Hence the probability of having all teams at low capability diminishes as 1/μn. In the public information case, the winner will take risks if the second ranked team is close to its own capability. Being close to a specific other team is inversely proportional to μ, and hence the probability of this diminishes as 1/μ. Our ability to manipulate the information levels of the teams or teams is likely to be somewhat limited, though we can encourage them to share more (in the low capability, low enmity situations) or alternatively to lock up their private knowledge5 (in situation of higher capability or enmity).

Finally, we can consider what happens when more teams are involved. Intuitively, competition might spur a race to the bottom if there are too many teams, and the model bears this out: in both the no-information and public information cases, adding extra teams strictly increases the dangers.

Conclusion. We’ve presented a model of AI arms race (which can be extended to other types of arms races and competitions) in which teams of AI designers can get ahead by skimping on safety. If the race takes some time there is a persistent ratchet effect (or race to the bottom) in which each team reduces their safety precautions slightly, until a Nash equilibrium is reached– possibly one of very high danger. Several factors influence on the probability of AI disaster, three in an intuitive way, and one counter-intuitively. Intuitively, if capability is much more important than the level of security precautions taken, the overall outcome is safer. Similarly, reduced enmity between the teams produces better outcomes, as does reducing the number of teams (in most situations). Counter-intuitively, increasing the information available to all the teams (about their own capability or progress towards AI, or that of the other teams) increases the risk. This is a special case of an information hazard [Bos11]: we’d be better offnot knowing. This model and variants of if may be of use in planning and regulating the development of AIs and other disruptive technological innovations.