Re: Friendliness SOLVED!

From: Thomas McCabe (
Date: Wed Mar 12 2008 - 21:17:45 MDT

On Wed, Mar 12, 2008 at 10:50 PM, Mark Waser <> wrote:
> >> Show me how I (or an AGI) can stay true to the declaration and still
> >> perform
> >> a horrible *and* unethical act OR
> >> Show me a set of circumstances where my Friendliness declaration prevents
> >> me
> >> (or an AGI) from protecting myself
> >
> > This is a false dichotomy. Neither me, nor other Singularitarians, nor
> > the AI, nor reality are obligated to choose between your two
> > predefined options.
> No! I mean that you can disprove my theory by providing a counter-example
> to either of the two OR any other counter-example that you can prove is
> relevant. I was trying to make life easier by showing where the best/most
> likely points to disprove my theory are.

I apologize for the misinterpretation.

> > How could this possibly be in the self-interest of, say, a paperclip
> > optimizer? It will obviously be able to create many more paperclips if
> > it ignores your (to the UFAI) funny-sounding pulses in fiberoptic
> > cables and just turns the Earth into a big pile of microscopic
> > paperclips.
> I handled the case of the paperclip optimizer in the e-mail that you are
> answering. To quote by pulling my previous statements from your e-mail
> >> For a powerful enough,
> >> single-goal entity that is sure that it *can* overcome all other
> >> entities,
> >> this is not going to stop it -- but this is a fantasy edge-case that we
> >> should be able to easily avoid.
> You *do* respond below by saying
> > Actually, it is probably the default case, and a large number of us
> > are operating off that assumption (it's the conservative scenario).
> > See
> So I'll just say OK, it doesn't handle this fantasy case (that can be easily
> outlawed by requiring that all AGI have a sufficient number of goals) but it
> has significant value outside this case (since I don't believe that either
> of us will convince the other).

We can't ban something as obviously destructive and difficult to
produce as nuclear-tipped ICBMs. How in Eru's name are we going to
enact a global ban on various kinds of AGI research? We have to build
a powerful FAI, which actually can ban dangerous AI research, before a
UFAI (or nuclear war, nanotech, etc.) kills us all. See, in addition to the
previous paper.

> >> I disagree. Prove me wrong by doing one of the two things above. That
> >> should be easy if my theory is as laughably wrong as you believe.
> >
> > By my count, four different people have now challenged your theory, so
> > there's plenty of other things to say. Stop repeating this
> > unreasonable demand; it isn't getting anyone anywhere.
> I've been saying a lot of things and four different people have challenged
> the theory but the truth is "NO ONE HAS PROVIDED A DETAILED, SPECIFIC
> UNHANDLED CASE OR A COUNTER-EXAMPLE" other than the paper-clip example which
> is an absurd edge case which we can simply outlaw by requiring all AGI to
> have a sufficient number of goals.

A sufficient number of goals is not enough; the vast, vast majority of
AGIs will then have a large set of unFriendly goals instead of a small
set of unFriendly goals. See

> If you feel that this statement is
> incorrect then I am asking you to provide what you believe is a detailed
> specific unhandled case or a counter-example. This is not an unreasonable
> demand. This is fundamental Bayesian logic. Show me some hard evidence
> that I am wrong. All I've seen thus far is just wild unfounded speculation.
> > Why does it even have this supergoal in the first place?
> Because that is the primary overriding goal in my Declaration of
> Friendliness. Why are you arguing with me if you haven't even had the
> courtesy to read what I have written well enough to realize this.

How will you get *every* AI, or *every* human, to endorse this
Declaration? And even if they do, how are you going to enforce it? How
do you guard against misinterpretations, or non-handled exceptions?

> > What other goals? An AI's goal system can be much, much simpler than a
> > human's. There's no reason why it has to have any other goals.
> True, but that is a fantasy edge case which we can easily outlaw.

 - Tom

This archive was generated by hypermail 2.1.5 : Wed Jul 17 2013 - 04:01:02 MDT