Re: Friendliness SOLVED!

From: Robin Brandt (
Date: Thu Mar 20 2008 - 07:46:47 MDT

Just answer this simple question?

Why would it be in the self-Interest of a Super-Intelligence to be Friendly
unless it already is friendly?
We all want AI to be friendly. But an AI has only the goals/values/interests
that we program into it. We have to code friendliness right there in the
core from the beginning. We cant just make a rational economic agent and try
to convince it that it is in itīs self-interest to be friendly by applying
some circular reasoning. Humans have built in moral circuity, but they donīt
work as platonic morality optimizers, they are tweaks that statistically
produce more offspring.
There is absolutely no rational reason to be friendly unless you have
certain values. Unless you value diversity, sentience, complexity, beauty,
individuality, subjectivity, preservation of information etc etc... The AI
will not be produces by evolution stumbling into pro-social area. It will be
built by humans that have. This is both tromendous advantage and a great
On Thu, Mar 20, 2008 at 1:19 PM, Nick Tarleton <> wrote:
> On Wed, Mar 19, 2008 at 10:20 PM, Mark Waser <> wrote:
> >  > Before you talk about defending a planet from a better-armed alien
> >  > invasion, here's a much simpler scenario: You have $1,000 worth of
> >  > stuff in your pocket. A robber walks up to you with a gun, and
> demands
> >  > that you hand over the loot. How will your Friendliness theory
> >  > convince the robber to leave, without shooting you or stealing your
> >  > stuff?
> >
> >  It probably won't.  He won't stick around long enough to be infected.
> >  Again, this is a strawman.  You're inventing a ridiculous scenario
> Ridiculous? This happens in reality ALL THE TIME.
> >  and
> >  demanding that I defend it.  I decline to do so unless you can show how
> it
> >  is at all justified or relevant.
> An UFAI need not listen to us long enough to be infected, either!
> So let's give you more time. Someone kidnaps you for ransom, ties you
> up in the basement, and for whatever reason lets you talk to them for
> a few hours. How do you convince them to be Friendly and let you go?
~Robin Brandt~
Emergence, Evolution, Intelligence
Control the infolepsy!
Love the world!

This archive was generated by hypermail 2.1.5 : Wed Jul 17 2013 - 04:01:02 MDT