Re: FAI prioritization

From: William Pearson (
Date: Thu Apr 03 2008 - 02:12:04 MDT

On 03/04/2008, Rolf Nelson <> wrote:
> For the duration of this thread, assume that FAI is the best use of
> time and resources for a rational altruist. How should resources be
> prioritized, in terms of marginal utility? Here are my current
> thoughts.
> My prioritization:
> 1. Outreach to mathematically-talented students and recent graduates.
> We know that some tiny minority of people exist who will self-motivate
> to help with FAI after a quick exposure to these talking points:
> * AGI may be possible within the next few decades
> * AGI can be dangerous. Suppose you initially give an AI the goal of
> making paperclips, with the plan that you will shut the AI down or
> modify its goals once you decide you have "enough paperclips."
> However, the AI would rather replace the Earth with paperclips
> according to its current goal, so it will spontaneously form a subgoal
> of preventing you from being willing and able to shut it off or
> rewrite its goals.

It is this that mathematically talented students are likely to find
silly in this sense

Unless they have read lots of sci-fi. Note note that AI may go wrong,
just that it will be affective enough to paper clip the world.


This archive was generated by hypermail 2.1.5 : Wed Jul 17 2013 - 04:01:02 MDT