From: Herb Martin (HerbM@LearnQuick.Com)
Date: Wed Nov 30 2005 - 22:02:35 MST
> "By the way, I believe that we will create friendly AI, but
> we will also (eventually) create unfriendly AI, either by
> accident or by design. "
>
>
> I'm just curious where you came up with that.
>
>
> By my understanding, these two are mutually exclusive... in
> the sense that
> if you actually create an FAI, it will quickly ensure that
> nobody create a
> UFAI (major existential risk), and if you create a UFAI
> first, then that
> tends to imply humans are, well, screwed.
>
Well, even assuming you are correct and that the first friendly
AI of sufficient interlligence eliminates the possibility of
an unfriendly AI (very doubtful but as least useful as a thought
experiment) then notice that SOMEONE would have created that
friendly AI, and someone else somewhere will be bound to disagree.
Do you expect everyone would agree that the US Military (or NSA,
or whatever) has the final say on "friendly", or that some
commercial organiation (e.g., IBM, Microsoft) has the final
say? Just look at the people who HATE Microsoft merely for
selling a lot of software that they don't wish to use.
Or imagine a terrorist or terrorist supporting nation has the
"first" friendly (in their terms) AI....
Or, it's open source? How do you stop some human from taking
it offline and tinkering with it until it is able to do battle
again with the Supreme Friendly AI running the world.
People cannot even agree on what would constitute 'friendly'
or 'unfriendly' behavior -- if it is powerful enough to prevent
any competitors it will by definition have to PROTECT itself
and, almost by definition, have to evolve to counter new threats,
at which point you cannot count on ANY pre-programmed friendly
behavior being permanent.
An evolving, all-powerful, friendly AI is a new species over
which we cannot guarantee control over any sufficiently long
period of time.
Even if we could guarantee this, we couldn't agree on the
definition of friendly or the evaluation of it's behavior
yesterday when it stopped some war between two countries.
Someone didn't get their way; perhaps someone had to die.
Someone does not think this is friendly. If you are lucky,
you are not on "that side."
-- Herb Martin
This archive was generated by hypermail 2.1.5 : Wed Jul 17 2013 - 04:00:53 MDT