Re: What are "AGI-first'ers" expecting AGI will teach us about FAI?

From: Samantha Atkins (
Date: Sun Apr 13 2008 - 01:41:13 MDT

Rolf Nelson wrote:
> On Fri, Feb 29, 2008 at 6:02 PM, Ben Goertzel <> wrote:
>> The fact that AGi ethics is incredibly badly understood right now, and
>> the only clear route to understand it better is to make more empirical
>> progress toward AGI. I find it unlikely that dramatic advances in AGI
>> ethical theory are going to be made in a vacuum, separate from
>> coupled advances in AGI practice. I know some others disagree on
>> this.
> For any of the many people who agree with Ben's sentiment:
> Large numbers of people have made various AI advances in the past. In
> none of these cases, to my knowledge, have FAI people said, "a-ha,
> that's one of the pieces of data I was waiting for, this advances FAI
> theory."
Because FAI was and is completely and utterly irrelevant to the level of
AI to date.

> Why would we expect this to change in the future? At the very
> least, doesn't this show that even if FAI advances require AGI
> advances, the "bottleneck" is that there are too few people working on
> deriving FAI from existing AGI, rather than too few people working on
> existing AGI?
There isn't enough progress in current AGI to derive any FAI from. So
why should any people be doing needlessly attempting such a thing?

> Are there specific facts about AGI that you're waiting to find out,
> such that if the result of a pending experiment is A, then successful
> FAI theory lies in one direction, but if the result is B, then
> successful FAI theory lies in a different direction? If so, what are
> such facts?
FAI isn't relevant if AGI does not exist.

> At what point will you know that AGI has advanced enough that FAI can proceed?
When it actually begins remotely to matter in that we have AGI that is
even at the level of a dog much less capable of doing us significant
harm. But it isn't a question of whether FAI can proceed but of
whether it is relevant and whether it can make any real progress.

> For SIAI specifically: how is OpenCog going to be "coupled" to
> "dramatic advances in AGI ethical theory"?

AGI ethical theory? What a fine intellectual abstraction. But that
seems to be all it is at this point.

- samantha

This archive was generated by hypermail 2.1.5 : Wed Jul 17 2013 - 04:01:02 MDT