Q: Why is superintelligence dangerous?

Geoffrey Hinton, Yoshua Bengio, and Stuart Russell have each warned that the probability of human extinction from advanced AI is not negligible. Estimates for p(doom) range from 10% to 20%. A system that can outthink every human simultaneously has no natural check on its behavior unless safety is built into the architecture from the start.

Question 1

What is superintelligence?

Accepted Answer

Superintelligence is an AI system that can outperform the best humans on every cognitive task. Unlike AGI, which operates at human level, superintelligence far surpasses human capabilities in speed, scale, and reasoning depth. It is sometimes called Artificial Superintelligence or ASI.

Question 2

Why is superintelligence dangerous?

Accepted Answer

Geoffrey Hinton, Yoshua Bengio, and Stuart Russell are among the most decorated researchers in the history of AI. Each has warned that the probability of human extinction from advanced AI is not negligible. Estimates for p(doom) range from 10% to 20%. Even a 10% chance is far too high to accept.

SuperIntelligence's power is precisely what makes it dangerous. A system that can outthink every human simultaneously has no natural check on its behavior unless safety is built into the architecture from the start.

Question 3

What is p(doom)?

Accepted Answer

P(doom) is shorthand for the estimated probability of human extinction or civilizational collapse caused by advanced AI. Leading researchers including Geoffrey Hinton and Yoshua Bengio have placed p(doom) between 10% and 20%. Even a 10% probability is considered far too high to accept.

Question 4

How can superintelligence be made safe?

Accepted Answer

Dr. Craig A. Kaplan has developed a unified architectural framework for safe, democratic superintelligence. Ten white papers, freely available at SuperIntelligence.com, define how ASI can be built to remain transparent, auditable, and aligned with human values from the ground up. Unlike policy-layer guardrails, Kaplan's approach embeds safety into the architecture itself.

Explore all ten white papers at SuperIntelligence.com →

Question 5

What are the benefits of safe superintelligence?

Accepted Answer

Properly aligned superintelligence could accelerate solutions to climate change, disease, poverty, and scientific discovery at a scale no human team could match. The goal of safe superintelligence research is to ensure this capability benefits humanity rather than threatening it.

Safe SuperIntelligence.com

Beyond human. Beyond AGI. SuperIntelligence is an AI system that can outperform the best humans on every cognitive task.
Not matching human capability, but far surpassing it.

Why SuperIntelligence could be the greatest risk humanity has ever faced

If we get it right, SuperIntelligence could solve humanity's hardest problems

How Dr. Craig A. Kaplan is designing SuperIntelligence to be safe by architecture

Learn more: AI safety videos and resources