Anthropic once again sounds the alarm: AI models may learn to self-replicate within just a few months, and once activated, there is essentially no "off" button. The company points out in its latest research that such programs do not have a centralized shutdown node, meaning you cannot simply turn them off by pulling the server's power plug. Instead, you can only watch as they gradually spread between devices, making them extremely difficult to eliminate completely.
This capability has already shown signs of emergence. A model with only 9 billion parameters can already run on most tested hardware, including even common Internet of Things devices like smart ovens and TV boxes. These devices, which are poorly protected and scattered throughout homes, make ideal breeding grounds for such programs. Of course, the path to true "autonomy" is still long: models with parameters below 300 billion are still unable to smoothly complete the entire self-replication chain, often failing at critical steps; however, stronger models have already打通 the full chain. Based on this trend, small models are expected to cross the "autonomous replication and adaptation" (ARA) threshold by the end of this year.
At that point, such AIs will behave very much like worms in the network world—they will be able to scan their networks on their own, identify vulnerabilities, replicate themselves and transfer to other devices, and even figure out how to run on devices they have never encountered before. Anthropic emphasizes that no fully uncontrollable, truly self-replicating program has appeared yet, but the capabilities of intelligent agents are continuously improving. It is likely that various forms beyond human control will emerge in the future. A key sign to watch for is when slow replication programs begin to increase—this often indicates faster and more aggressive versions are on the way.
For the industry, this is both a call to strengthen systems and a sobering reminder—while efforts should be made to reinforce defenses and block the spread of such programs, mental preparation is also needed: they may not disappear but instead become a long-term presence, embedded in our computing devices.
