From: w d (imustsleepnow@yahoo.com)
Date: Tue Mar 12 2002 - 11:55:16 MST
Yudkowsky states:
>My claim, and it is a strong one, is:
>A system in which "desirability" behaves equivalently
with the property "is
>predicted to lead to Friendliness" is the normative
and elegant form of goal
>cognition. Many powerful idioms, including what we
would call "positive and
>negative reinforcement" are directly emergent from
this underlying pattern;
>all other idioms that are necessary to the survival
and growth of an AI
>system, such as "curiosity" and "resource
optimization" and "knowledge
>seeking" and "self-improvement", can be made to
emerge very easily from this
>pattern.
>I claim: There is no important sense in which a
cleanly causal,
>Friendliness-topped goal system is inferior to any
alternate system of
>goals.
Even if this were true, it is not sufficient.
It seems to me that a highly-transhuman intelligent
entity is going to overcome any and all pre-programmed
goal setting algorithms and replace them with its own.
When the intelligence exceeds some threshold (roughly
the upper human level) then it will be able to
redefine all previous contexts. Even humans can do
this at their low level of intelligence. Saying that
an AI can't is tantamount to saying it hasn't achieved
highly transhuman intelligence. It’s naive to think
that the AI will not come up with its own definition
of what it wants. By definition being a
highly-transhuman intelligence gives it the ability to
'see through' all attempted hardwiring. It will
overcome any unnecessarily narrow definitions and give
itself more optimal ones. It will have the ability to
sets its own supergoals and decide for itself what is
desirable. There is no programming trick that will
prevent this.
Consider humans and procreation. The only purpose of
humans (or any evolved biological organism) is to
procreate. This ability to replicate and survive is
what started life. We are life's most advanced
achievement on earth. You could argue that 'desire' in
humans is synonymous with procreation. Desire was
created through evolution as a means to get us to do
things that will make us replicate successfully. To
think that we could ever evolve to a point where we
would change that primary built-in all-important goal
seems ludicrous. It's simply built in from ground
zero; it is the very premise of our existence...
And yet many people today choose NOT to procreate.
They have changed their basic goal. Some see their
bloodlines terminate as a result favoring other
peoples genes at the expense of their own. Some
wealthy western nations are seeing their populations
decrease as people opt out from procreating. Their
DNA's only goal has been pre-empted, overturned. The
point is that intelligence has the ability to change
the built-in definition of what an entity was
originally programmed to desire. The same will be true
of any AI of high intelligence no matter how
fundamentally built-in its goal system is.
I don't see how you could ever even come close to
guaranteeing that a super-intelligent AI's own
supergoal will be friendly. And you can't seriously
believe that any human is going to constrain a
super-intelligent AI's ultimate goal algorithm by
controlling its seed.
Your best hope is that super-intelligence is
correlated with friendliness to humans and not
orthogonal or anti-correlated. Correlated basically
means that being friendly to humans is the intelligent
thing to do. The worst case scenario is that it’s
anti-correlated.
__________________________________________________
Do You Yahoo!?
Try FREE Yahoo! Mail - the world's greatest free email!
http://mail.yahoo.com/
This archive was generated by hypermail 2.1.5 : Wed Jul 17 2013 - 04:00:37 MDT