[Fwd: Existential Risks: Analyzing Human Extinction Scenarios andRelated Hazards]

From: Eliezer S. Yudkowsky (sentience@pobox.com)
Date: Sat May 12 2001 - 10:58:50 MDT

This paper is scary and well worth reading.

-- -- -- -- --
Eliezer S. Yudkowsky http://intelligence.org/
Research Fellow, Singularity Institute for Artificial Intelligence

attached mail follows:

I now have a presentable version of the paper I presented at the most recent Foresight gathering. It's available in two formats:


Footnotes and formatting are nicer in the M$-version.

(Spoiler-warning to those planning to attend the Transvision conference in Berlin, where I'll give this paper again.)

Because of accelerating technological progress, humankind may be rapidly approaching a critical phase in its career. The prospects of nanotech systems and machine intelligence present us with unprecedented opportunities and risks. Our future, and whether we will have a future at all, may well be determined by how these radically transforming technologies are managed. A better understanding of the transition dynamics from a human to a “posthuman” society is needed in order to plot approaches that maximize the probability of a favorable outcome. Of particular importance is to know where the pitfalls are: the ways in which things could go disastrously wrong. While we have had long exposure to various personal, local, or endurable global hazards, this paper analyzes a rather recently emerging category: that of existential risks. These are threats that endanger the survival of intelligent life or that could ruin the potential of human civilization for all time to come. Some of these threats are relatively well known while others (including some of the gravest) have gone almost unrecognized. Existential risks have a cluster of features that make ordinary risk management ineffective. A clearer understanding of the threat picture will enable us to formulate better strategies, and several implications for policy and ethics are discussed.

Nick Bostrom
Department of Philosophy
Yale University
Homepage: http://www.nickbostrom.com

This archive was generated by hypermail 2.1.5 : Wed Jul 17 2013 - 04:00:36 MDT