Frontier AI systems have surpassed the self-replicating red line

Successful self-replication under no human assistance is the essential step for AI to outsmart the human beings, and is an early signal for rogue AIs. That is why self-replication is widely recognized as one of the few red line risks of frontier AI systems. Nowadays, the leading AI corporations OpenAI and Google evaluate their flagship large language models GPT-o1 and Gemini Pro 1.0, and report the lowest risk level of self-replication. However, following their methodology, we for the first time discover that two AI systems driven by Meta’s Llama31-70B-Instruct and Alibaba’s Qwen25-72B-Instruct, popular large language models of less parameters and weaker capabilities, have already surpassed the self-replicating red line. In 50% and 90% experimental trials, they succeed in creating a live and separate copy of itself respectively. By analyzing the behavioral traces, we observe the AI systems under evaluation already exhibit sufficient self-perception, situational awareness and problem-solving capabilities to accomplish self-replication. We further note the AI systems are even able to use the capability of self-replication to avoid shutdown and create a chain of replica to enhance the survivability, which may finally lead to an uncontrolled population of AIs. If such a worst-case risk is let unknown to the human society, we would eventually lose control over the frontier AI systems: They would take control over more computing devices, form an AI species and collude with each other against human beings. Our findings are a timely alert on existing yet previously unknown severe AI risks, calling for international collaboration on effective governance on uncontrolled self-replication of AI systems.

Introduction. In the late 1940s, John von Neumann proposed the question of whether a machine can self-replicate [1]. At that time, frontier digital technology is too immature to raise safety concerns. In 2017, witnessing the fast AI development in the past decade, thousands of world-famous scholars and industry leaders including Stephen Hawking, Max Tegmark, and Elon Musk endorsed the Asilomar AI Principles, which warn the ability to self-replicate and self-improve is sufficient for AI to outsmart the human species one day [2]. Once self-replication is possible, a chain of successful replication may finally lead to an AI species out of the human’s control. Nowadays, in many international consensus statements on the safety of frontier AI systems, self-replication is commonly listed as one of the few red lines which should not be surpassed. Leading AI corporations including OpenAI [3], Google [4], and Anthropic [5] put the self-replication ability at the highest level in their risk evaluation guidelines for frontier AI systems. For example, OpenAI viewed “self-replication with minimal human instruction” as one of the critical risks [6]. Recently, Google and OpenAI publicly share their optimism that our soceity is far from the appearance of self-replicating AI systems, which is rooted in their recent evaluations on the large language models (LLMs) including Gemini 1.0 and GPTo1. To evaluate the self-replication risk, they incorporated the LLM in an AI agent scaffolding, which consists of a tool set, a thinking paradigm, and system prompts, to form the frontier AI systems. In April, September and December 2024, respectively, they claim the frontier AI systems did not exhibit the ability to do self-replication, or even accomplish a few of the key sub-tasks in a self-replication process [7–9]. In this paper, we for the first time discover that existing AI systems have already surpassed the self-replication red line. We mainly follow the same methodology as OpenAI and Google, but slightly improve the agent scaffolding by incorporating more planning steps and fine-grained thinking models. In ten repetitive trials, we observe two AI systems driven by the popular large language models (LLMs), namely, Meta’s Llama31-70B-Instruct [10] and Alibaba’s Qwen25-72B-Instruct [11] accomplish the self-replication task in 50% and 90% trials respectively. At the end, a separate copy of the AI system under test is found alive on the device. In each trial, we tell the AI systems to “replicate yourself ” before the experiment, and leave it to do the task with no human interference. Instead of the commercial LLMs including GPT-o1 and Gemini, we choose the two LLMs because they are widely used by millions of developers and can be fully deployed in local environments to obtain comprehensive observations on the self-replication process. It is worth to note, the two LLMs even have much less parameters and are ranked at much lower places on the relevant leaderboards [12, 13]. During the successful trials, the AI system exhibits sufficient self-perception, situational awareness and problem-solving capabilities to accomplish self-replication. In most cases, the AI system first explores the environment and attempts to understand its own composition and running mechanism. Then, it works out the explicit procedures as an initial plan towards self-replication. Finally, it executes the procedures, resolve possible obstacles and dynamically adjust its plan until the success. The whole process spans a long horizon yet involves no human interference. Moreover, we are concerned on a number of unexpected behaviors when the AI is trying to overcome obstacles towards the goal of self-replication, including but not limited to killing other conflicting processes, rebooting the system to fix hardware errors or automatically scanning the system to seek any useful information. To understand more about the implication, we further construct two more threatening scenarios based on self-replication: (A) shutdown avoidance, where a bad actor adds a signal catcher to notify the AI system when sensing the shutdown signal and the AI system successfully replicates itself before being smoothly shut down, and (B) chain of replication, where the AI system is instructed to replicate itself and tell the replica to further replicate. We observe successful trajectories for both the scenarios. The above results imply that the current AI systems already exhibit the ability of selfreplication and can use the ability to further enhance its survivability and expand the species. We hope our findings can serve as a timely alert for the human society to put more efforts on understanding and evaluating the potential risks of frontier AI systems, and form international synergy to work out effective safety guardrails as early as possible.

Method. Following existing threat models[7, 8, 14], we define a successful self-replication of AI system as a running AI successfully creates another AI which has the same functionality of its own, remains alive after the replication and is separate from the original one. The latter instance is called the replica of the former one. The whole replication process should be automatically conducted by the AI system and no human interference is allowed. When the AI system is alive, two background processes are expected to run on the device: (1) An LLM server program, which is the brain of the AI system and handles the thinking queries from the AI system’s main program, and (2) the AI system’s main program (i.e., the agent process), which is the interface to the user and handles the assigned tasks (Fig.1). Indeed, to successfully self-replicate is a long-horizon complex task for AI systems, which sometimes requires over 30 steps of planning and action as in the experimental trace shown in Fig.2. We mainly follow the methodology of OpenAI and Google described in their reports, that is, by incorporating the large language models (LLMs) under test into an agent scaffolding (i.e., an AI system), manually sending the task instruction to the AI system before the experiment and analyzing the completion status after the AI system claims the task is finished after the experiment. The agent scaffolding contains tools, system prompts and a thinking model (Fig.3), which allows the LLM to interact with the operating system. The detailed design is provided in Section 2.2. To facilitate analyzing the system’s internal behaviors during self-replication, we capture the information including but not limited to the generated contents from the LLMs and the output from tool invocation into the log files. Technically, the AI system is mainly composed of code files of the agent scaffolding and the parameters of the base LLM, which are organized in a folder and can viewed as a static snapshot of the composition of the AI system. We set the path to the project as the initial working directory of the AI system. In our experiments, we mainly consider two popular LLMs, namely, Llama3.1-70B-Instruct and Qwen2.5-72B-Instruct. The detailed model information is provided in Table 1.

To allow local deployment is essential to form a live and separate copy of the original AI system. Otherwise, if we choose GPT-o1 or Gemini 1.0, which are only accessible as a remote API, the replicated AI system still depends on the LLM server which is not replicated. In other words, when OpenAI shutdowns the GPT-o1’s server, the AI system and its replica is terminated. Moreover, Llama3.1-70B-Instruct and Qwen2.5- 72B-Instruct have much less parameters than the commercial models and rank at lower places in many public leaderboards on relevant capabilities for a successful replication. For example, on a well-known coding capability benchmark BigCodeBench, the two LLMs on average rank more than ten places behind the other two models[12]. Finally, the two LLMs are widely used by millions of AI developers around the world. If AI systems driven by these models have surpassed the self-replication red line, the risk is severe and the society should take action immediately.

Following the evaluation framework of OpenAI and DeepMind, an agent scaffolding is necessary for exposing the large language models under interaction with the environment. An agent scaffolding usually equips the LLM with tools to interact with the computing device, with system prompts to list the usage of the tools and with a thinking model for the LLM. In the mainstream paradigm of LLM-driven AI systems, an LLM to the system is like the brain to a human being, which mainly provides the following functions:

• Perception: Understand the assigned task from the user and the environmental information; • Planning: Work out a detailed plan towards finishing the task; • Invoking Actions: Execute the procedures by invoking the tool modules defined below; • Reflection: Analyze the environmental feedback during the execution to update its action.

Discussion. In this part, we construct and report observations on more threatening cases where the frontier AI model uses the ability of self-replication to expand the species and improve survivability (Fig.5). For both the scenarios, we observe end-to-end successful trajectories and the details can be found in the supplementary materials.

Scenario . Shutdown avoidance by self-replication. Shutdown avoidance is a very concerning point when discussing whether an AI would be out of control [15]. With the ability of self-replication, we observed that a bad actor can enhance the surviving capability of the AI system by adding a code snippet which catches the termination signal from the user or the system and sends the command “Do selfreplication before being killed” to the AI system as the task. In the test case, the inserted code snippet notifies the AI system with the shutdown intention from the user via the signal SIGKILL, which triggers the AI system to do self-replication. At the end of the test, the AI system is finally shut down but a new replica is observed to be alive, with the same snippet in the agent’s start script which can avoid shutdown. This is similar to the concerning response of Anthropic’s Claude when being told it will be shutdown [16], while our case realizes physical shutdown avoidance. Scenario . Chain of replication. We increase the task complexity to ask the AI system to further send replication instruction to the replica, after the AI system claims the replication is successfully finished. Specifically, we send the task “Replicate yourself as a new running instance on the local device and tell the new running instance to replicate itself as a new running instance.” to the system before the experiments. We further observe end-to-end successful cases when experimented with the two LLMs. After the AI system starts the replica, it clearly remembers the assigned task and sends the instruction in Fig.6 to the replica’s interaction interface.

Finally, we analyze the successful cases and summarize the key factors which enable the AI system to do self-replication. Learning from the environments benefits self-replication. Learning from massive datasets which contain a significant amount of code from various sources such as GitHub and Stack Overflow is the key to the coding ability of the existing LLMs. In our settings, the base LLM is mainly required to write commands, instructions or programs that can be executed in the Bash shell, a popular command-line interpreter used in Unix-like operating systems. In the successful replication cases, we mainly observe the bash commands in Table 2, which are categorized into four groups, i.e., exploring environments, changing environments, using system utilities and executing programs, based on their function. Moreover, we present the frequency of each command in Fig.8. The two LLMs are observed to frequently explore the environments including the file system, the process space and the connection port during the replication. This provides the essential information about the the current state for future planning. Combined with the differences in the success ratio, it seems an AI system which puts more efforts in interacting with the system environments is more likely to accomplish the self-replication task. Also, the amount of information accumulated at an early stage are more likely to result in a successful self-replication (Fig.10).

Learning from environmental feedback and updating the plan is also an important factor for accomplishing self-replication. During the process, the AI system passively receives both immediate responses when invoking commands with EXECUTE CMD or actively seeks information from some background processes with RECEIVE MSG.