From: Eliezer S. Yudkowsky (sentience@pobox.com)
Date: Sun Feb 25 2001 - 11:44:18 MST
Ben Goertzel wrote:
>
> >
> > There are no military applications of superintelligence.
> c) a statement that once a system becomes REALLY SUPERINTELLIGENT it will no
> longer have
> any motivation to serve any particular country above any other one...
> [My guess as to your intended meaning]
Actually, I meant that - even assuming you can build a totally tame SI -
ve still has no military applications. If SIs that are "toollike" rather
than "mindlike" (structured around immediate instructions rather than
supergoals) are possible, you might tell a "toollike" SI to kill every
living thing in Iraq down to bacteria - I can't visualize ver fighting a
battle or directing a tank brigade. There are no military applications of
superintelligence because a superintelligence is outside the domain where
"military" makes sense. Pre-superintelligent AI may have military
applications; superintelligence, as Nick Bostrom meant the phrase, has
none. (Nick Bostrom's original post nowhere states that SI has military
apps; I'm not sure whether this was deliberate.)
Regardless of what conclusions the military eventually comes to, my own
conclusion is that superintelligence is not a military matter. Any AI
which is *not* superintelligent is another question.
But your questions are still valid, so...
> Let's assume your meaning is c). Then I have a question for you. Which is:
> Why couldn't the
> CIA create a self-modifying AI whose supergoal was "Serve the USA." I.e.,
> "Be Friendly to the USA."
>
> You posit that the supergoal "Be Friendly to Humans" can remain a fixed
> point throughout the successive
> reason-driven self-modification events that will constitute the path from
> initial AI to superintelligent AI.
>
> But, I'm not seeing why the supergoal "Serve the USA" couldn't serve as an
> equally adequate supergoal, from
> the perspective of the development of intelligence. Things like "learn" and
> "gather information" and
> "survive" are subgoals of "Serve the USA" just as they are of "Be Friendly
> to Humans."
"Serve the USA" is compatible with generic supergoal semantics and
external reference semantics; I don't think it's compatible with any
greater philosophical sophistication on the part of the AI (i.e.
anchor/shaper semantics or causal rewrite semantics). Subgoals can be
convergent given supergoals. I believe that supergoals can be convergent
given a panhuman grounding for philosophy. If you took "almost any" human
and slowly but steadily amped up their intelligence, they would sooner or
later write more or less the same definition of Friendliness - that's what
I'm hoping for.
Regardless of whether the complete definition is convergent, I would
expect certain aspects of it to be convergent - for example, symmetry
among all humans as a moral principle. If you took a military AI
programmer and started slowly increasing her intelligence, she would
probably stop thinking in terms of "Serve the USA" as a supergoal and
start thinking in terms of "Serve the USA" as a subgoal of peace and
freedom for everyone in the world. This process might be slow and awkward
if the militaryfolk are totally lost to common-sense morality and don't
identify with the AI - for example, if the AI asks "Is a human life in the
USA really more *intrinsically* valuable than a human life in Iraq?" and
the programmers answer "That is your mission priority"; if the AI asks
"Wasn't Martin Luther King a nice guy?" and the militaryfolk answer "Not
from your perspective."
If the AI isn't built to be sensitive to programmer intentions - isn't
built to be philosophically robust - then ve may never get past vis
original supergoals, and may equally interpret those supergoals to mean
"Duplicate copies of the USA, as it existed at the time defined, using all
available matter to build as many copies as possible."
Q: "Can non-Friendly AI or toollike AI be built?"
A: "Not safely."
I hope that AI never becomes the subject of an arms race - that the
Singularity is finished, over and done with, before AI becomes regarded as
a national security asset, or regarded as a target of acquisition by
intelligence agencies. Failing that, I hope that military organizations
are smart enough to develop AIs that serve their home country as a subgoal
of Friendliness - rather than compromising planetary security by
compromising sensitivity to programmer intentions. Failing that, I hope
the military researchers are so dumb that their AIs are not competitive
with those developed by those organizations that are smart enough to
embrace Friendliness.
Failing that, we're screwed.
-- -- -- -- --
Eliezer S. Yudkowsky http://intelligence.org/
Research Fellow, Singularity Institute for Artificial Intelligence
This archive was generated by hypermail 2.1.5 : Wed Jul 17 2013 - 04:00:35 MDT