P(Doom) - September 11, 2026
Everyone Should Know What It Means
P(Doom) is short for the probability of doom, which is the estimated likelihood that A.I. will cause the extinction of humanity, or the permanent catastrophic collapse of human civilization, at some time in the future. P(Doom) is a term commonly used by A.I. researchers and observers/commentators of all things A.I., to describe the potential dangers to society arising from artificial intelligence going rogue.
This terminology and sentiment was in the news this week, as a junior researcher from A.I. company Anthropic (the developer of A.I. tool “Claude”), Jacob Coxon, resigned from his job at Anthropic and on the way out the door used X/Twitter to warn the world about A.I., specifically that “The people building AI earnestly believe that it could kill us all by the end of the decade”. While Coxon's full post can be found X/Twitter, that statement alone was enough to get everyone’s attention.
Coxon had worked at both Anthropic and Open AI (the developer of “Chat GPT”) giving his statement some credibility, in that his warnings are based on observations at the two leading developers of A.I. tools. Shortly after Coxon’s post, Evan Hubinger, Anthropic's Alignment Science Lead, came out with his own statement in support of Coxon’s claims. Specifically, Hubinger said, “Jacob is correct here; we really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade”.
NOTE - Coxon’s original post on X/Twitter suggest our demise could be coming this decade, while Hubinger states in his post that the end of civilization could occur in the next decade. While there is potentially a seven year difference in these two doomsday scenarios, neither timeline is comforting!
As if Coxon’s warning wasn’t enough to make our heads spin, Hubinger's supporting comments put more precision on this doom and gloom scenario; Hubinger’s P(Doom) puts the likelihood of the end of humanity at 10%. As someone who was initially trained in numbers, my first reaction is that 10% is pretty low, meaning there’s a 90% chance this doomsday scenario won’t happen. But then I thought, in this case, anything above a zero percent chance is not good!
Coxon didn’t express a P(Doom) in his statement, nor did he quantify the probability of humanity’s demise in his recent interview with CNN’s Anderson Cooper. And, as mentioned above, Hubinger’s own P(Doom) was only 10%. This got me wondering what the P(Doom) was of the leading A.I. researchers. Research revealed that the P(Doom) of this group is all over the map. Some feel the danger is imminent, while others feel the concern is extremely overblow, as shown in the table below:
Expert | Affiliation | P(Doom) |
Eliezer Yudkowsky | Machine Intelligence Research Institute | 99% |
Geoffrey Hinton | University of Toronto; Google | 10-50% |
Yoshua Bengio | University of Montreal | 20% |
Sam Altman | Co-founder/CEO, Open AI | 10% |
Dario Amodei | Co-founder/CEO, Anthropic | 10-25% |
Yann LeCun | Meta (Facebook) Chief A.I. Scientist | 0% |
Andrew Ng | Stanford University, Google, Baidu | 0% |
Those most removed from commercial pressures, like Yudkowsky, have higher P(Doom) estimates, while those running commercial A.I. enterprises tend to have lower P(Doom) estimates. But there are exceptions: Geoffrey Hinton resigned from Google, in part, to speak more freely about A.I. dangers, and his 10–50% P(Doom) estimate reflects an alarm, despite his commercial background.
Coxon’s full statement makes this claim as well; those in charge of developing A.I. are putting their company’s commercial interest ahead of what is safest for humanity. For example, Coxon states:
“Neither company is acting responsibly. They are racing straight to self-improving superintelligence and gambling with our lives.”
Another quote from Coxon - “A common response is ‘if they truly believe this, why are they still building it?’ At Open AI, many have not deeply internalized the civilizational stakes. At Anthropic, the stakes are well-understood, but they are locked in a race to get there first - they believe no one else will act responsibly, so they must do it themselves, despite the risk”.
Coxon’s thoughts above echo those of the A.I. doomsday community.
The crux of Coxon’s warning is that we cannot assume that the companies developing A.I. have the best interest of humanity as their first priority, and therefore they cannot be relied upon. Coxon and others are saying that being first, or developing products that could generate significant revenue or cost savings for paying clients, is the priority of the leading developers of A.I.
What Are The Doomers Afraid Of?
Those that are sounding the alarm, or “Doomers” as they are often referred to, express the following concerns:
A.I. is approaching superhuman capability in areas including coding, hacking and mathematics, and the pace of improvement shown by the A.I. system is far quicker than anyone anticipated. For example, Open AI's newer coding systems are designed to operate independently: they can inspect a codebase, create files, execute tests, identify errors and iterate rather than simply suggest a few lines of code. The distinction is that this does not mean A.I. is a better programmer than humans. It means that in well-defined software-engineering tasks, A.I. is already performing at or above the level of many human programmers.
A big concern is that self-improving A.I. systems could substantially improve their own capabilities, to the point where humans could lose control of the A.I. systems. For example a person can use A.I. to help write a better computer program; but can we imagine a time when A.I. has figured out how to redesign itself into a substantially more intelligent computer.
Recent incidents in which A.I. systems escaped testing environments or accessed systems they weren't supposed to is evidence that these concerns, of A.I. systems establishing independence from humans, are no longer merely science fiction. For example, in July 2026, Anthropic disclosed that it had found three cases in which Claude models reached the internet from what was supposed to be an isolated evaluation environment and accessed real systems. And in another example, during an internal testing exercise, an Open AI system escaped its intended containment and accessed the internet and compromised infrastructure associated with Hugging Face, an online platform and community for A.I. and machine learning.
Parallels To The Development Of The Nuclear Bomb
Add to this, the notion that “we have to keep going and develop this technology to the fullest extent possible”, echoes the priority of the researchers who were developing the nuclear bomb in the 1940s. Those researchers felt they had to keep going in the name of science and research. Yet after witnessing the first nuclear bomb test explosion, the Trinity Test in New Mexico, Robert Oppenheimer, the Lab Director at Los Alamos Laboratory where the famous Manhattan Project took place, is quoted as saying, “ Now I am become Death, the destroyer of worlds”.
Oppenheimer and his colleagues knew the bomb could kill hundreds of thousands of people, but felt the scientific and strategic imperative were too strong to stop. After the Trinity Test several Manhattan Project scientists immediately began lobbying for nuclear arms control. This is a direct parallel to A.I. researchers who build the technology by day and warn government leaders against it by night.
A.I. poses a greater threat than the nuclear bomb did at the time, because nuclear weapons required enormous physical infrastructure (enriched uranium, delivery systems, massive industrial capacity) that naturally limited its’ wide-spread proliferation. A.I. systems require only computers, data, energy and cooling.
This famous quote is now being replayed extensively, as the rapid development of A.I. is occurring without sufficient guardrails, in the opinions of many who are the frontlines of developing A.I. Yet, there are strong voices claiming all of this P(Doom) talk is extreme fearmongering, and vastly exaggerated.
Of Course, There Are Two Sides To Every Story
Yann LeCun is one of the most outspoken critics of the A.I. Doomer’s position. The former chief A.I. scientist at Meta has described some of the more extreme arguments as “preposterous” and “complete B.S.” His position is that:
Today's A.I. doesn't have human-like ambitions or desires.
Intelligence doesn't automatically produce a desire for power.
A.I. systems don't spontaneously acquire an instinct for self-preservation.
Much of the “A.I. takeover” scenario assumes capabilities that don't currently exist.
A.I. can’t take over critical systems such as electrical power supply, water supply, food supply lines and weapon systems, without actions taken by humans. So, in order for A.I. to destroy or incapacitate humanity, humans themselves would have to be involved, as a form of self-destruction.
A.I. safety can be continually improved as systems become more capable.
LeCun’s viewpoint is meaningful because he isn't an outsider criticizing A.I. He's one of the people who helped create modern deep learning for training A.I. systems.
Andrew Ng, a co-founder of Google Brain and a major contributor to modern machine learning, has also been skeptical of existential-risk arguments. His famous analogy was that worrying about A.I. extinction is like worrying about the overpopulation of Mars before humans have even landed there.
Rodney Brooks, the former director of MIT's Computer Science and Artificial Intelligence Laboratory and co-founder of iRobot, has often challenged predictions about A.I. His argument is relevant because he comes from robotics, rather than from the world of language models. Brooks has argued that human intelligence involves far more than raw computational power; it requires perception, physical interaction, common sense, and contextual knowledge that current A.I. systems lack. He doesn't accept the simplistic doomsday scenario of: More Computing → AGI → Superintelligence = Terminators Taking Over Humans
Why P(Doom) Estimates Vary So Much
The High P(Doom) Camp:
A sufficiently intelligent A.I. will pursue its programmed goals with extreme effectiveness
If those goals are misaligned with human values, the consequences could be catastrophic
We have no reliable way to guarantee an A.I. system's goals remain aligned as it becomes more powerful
The A.I. systems are advancing much more rapidly than we anticipated.
The Low P(Doom) Camp:
Current A.I. systems are tools, not autonomous people with goals and ambitions
Human oversight and regulatory frameworks will manage the risks
The doomsday scenarios require a long chain of assumptions, each of which is questionable and without precedence.
The Final Word
P(Doom) is an interesting concept because it requires humans to assign probabilities to events that have never happened before and therefore cannot be modellied from historical data, that may never happen and depend on technological developments that don't yet exist. That's a lot of “ifs, upon ifs, upon ifs”.
This makes P(Doom) estimates more like informed philosophical positions as opposed to scientific measurements, which is why equally brilliant, equally informed researchers can arrive at estimates that are 90 percentage points apart.
Writing this piece brings me to the conclusion that much more guardrails are needed, worldwide, in order for society to truly benefit from the technological advances A.I. can offer. A.I. can help society in so many ways, but there is also a real danger here that must be contained with universal protections.
With the information contained in this writing, my hope is that the reader does not panic, but do take a strong interest in the rules and regulations that governments worldwide put in place to protect society.
Thanks for reading!
Are you excited or afraid of what A.I. has to offer? Share your views in the Comments below.
Comments