If you've read any AI news in the past week, you've seen a strange little term everywhere: P(doom).
Anthropic's alignment lead said his personal P(doom) is "more than 10% within the next decade." Skeptics called the number "made up." Confusion everywhere. So let's actually explain it — simply, without the hype.
What Is P(doom)?
P(doom) is shorthand for "the probability of doom" — specifically, an individual's subjective estimate of the chance that advanced AI causes a catastrophic outcome for humanity, up to and including human extinction.
Breaking down the name:
- P = probability (a number between 0 and 1, or 0% and 100%)
- doom = the bad outcome — usually meaning AI causing human extinction or an irreversible global catastrophe
So "my P(doom) is 10%" translates to: "I think there's a 1-in-10 chance advanced AI ends really badly for humanity."
It's not a formula, a benchmark, or a measurement. It's a personal belief expressed as a number. That's the single most important thing to understand.
Where Did the Term Come From?
The concept grew out of the rationalist and effective altruism communities in the 2010s, particularly discussions around future AI systems. Researchers who worried about advanced AI needed a compact way to compare how worried they were. Instead of writing essays, they could just say a number.
For years, P(doom) lived in niche forums and conference hallway conversations. Then, in the past week, it went fully mainstream when a senior Anthropic researcher put a number greater than 10% on it in public, following a high-profile resignation from the company over exactly these concerns.
Why Do Expert Estimates Range From 0.1% to 90%?
Surveys of AI researchers have produced estimates spanning several orders of magnitude. The reasons for the disagreement are instructive:
1. There's no data to extrapolate from
We've never built a system smarter than humanity. There is no reference class — no historical record of "previous times a species built a superior intelligence." Statisticians hate extrapolating from a sample size of zero.
2. "Doom" isn't precisely defined
Does doom mean extinction? Permanent civilizational collapse? A locked-in dystopia? Mass unemployment and unrest? Depending on who you ask, the event being estimated is completely different — so of course the numbers differ.
3. It depends on contested assumptions
Your P(doom) depends on what you believe about:
- How fast AI capabilities will improve (slow increments vs. sudden jumps)
- Whether alignment techniques will scale (can we reliably control systems smarter than us?)
- How coordinated labs and governments will be (race dynamics vs. careful pacing)
- Whether dangerous capabilities arrive gradually (with warning signs) or suddenly
Change any assumption and the number moves dramatically.
4. Psychology and incentives are in the mix
Cynics point out that a high P(doom) raises the perceived importance of your own work — and of your company's models. Believers respond that many researchers have left lucrative jobs over these convictions. Both things can be true.
How to Think About P(doom) Rationally
You don't need to pick a side. Here's a saner framework:
Treat it like insurance math, not prophecy
Nobody knows whether your house will burn down. We still buy insurance based on the possibility and the severity. Similarly, the useful question isn't "is P(doom) exactly 10%?" but "if there's even a small chance of an irreversible catastrophe, what cheap precautions are worth taking?" — things like evals, safeguards, watermarks, and coordinated pacing.
Distinguish calibrated estimates from rhetoric
A number with no reasoning behind it is a vibe, not a probability. When someone quotes P(doom), ask: what's the decomposition? What would change their mind? If there's no answer, treat the number as an expression of emotion.
Separate tail risk from base-rate thinking
Some outcomes are low-probability but so severe that expected-value calculations still justify attention. "It probably won't happen" and "we should take it seriously" are compatible statements — in the same way "your flight probably won't crash" coexists with entire industries devoted to making crashes less likely.
The Current Debate in One Paragraph
This week's firestorm started when researcher Jacob Coxon resigned from Anthropic saying labs are "gambling with our lives," and Anthropic's alignment lead amplified it with a ">10% within the decade" P(doom). Critics replied that the number is uncalibrated, possibly a "flex" to advertise model capability, and awkward for a company heading toward an IPO. CEO Dario Amodei then published a proposal for pacing frontier development. Everyone is arguing about the number; almost nobody is arguing about the reasoning behind it — which is exactly the part that matters.
Key Takeaways
- P(doom) = one person's subjective probability that AI causes catastrophe. Not a measurement.
- Expert estimates vary wildly because there's no data, no shared definition, and no agreed model.
- The useful move is insurance-style thinking: small probability × enormous severity justifies cheap precautions.
- Ask for reasoning, not numbers. "10%" means nothing without the assumptions behind it.
Want to go deeper on AI safety concepts? Check out our explainers on what is post-training and what is speculative decoding — or explore the actual models at the center of this debate on Qubax's model marketplace.
FAQ
Is P(doom) a scientific measurement?
No. It's a subjective probability estimate — a belief expressed as a number. There is currently no scientific method to calculate the probability of AI catastrophe with any precision.
What is a "normal" P(doom) to have?
There isn't one. Large surveys of ML researchers show medians in the low single digits for extinction-level outcomes, but with enormous variance — and many leading researchers sit far above or below the median.
Does a high P(doom) mean we should stop developing AI?
That's the active policy debate. Anthropic's CEO recently proposed pacing rather than stopping frontier development — coordinated slowdowns at capability thresholds. Others argue the benefits (medicine, science) justify proceeding carefully. There's no consensus.
Why should a developer care about any of this?
Because it shapes regulation, model access policies, and safeguard features that directly affect what you can build. The practical impact lands on your API keys long before it lands on humanity.
