From: Michael Wilson (mwdestinystar@yahoo.co.uk)
Date: Wed May 18 2005 - 15:27:21 MDT
> I believe that any project in which cognition is a possible outcome
> should be monitored at the very least
Sure, thanks for volunteering. :)
> and approached should it appear that progress is being made.
Unfortunately the prognosis for this isn't good. Most researchers
simply don't see the problem and tend to refuse to do so when
confronted with the possibility. The basic reason for this is that
takeoff and unfriendliness are counter-intuitive and have
unpleasant implications, both in terms of risk and the amount of
work the researcher would have to do. AGI researchers who didn't
start with a good idea of the necessity and difficulty of
Friendliness will almost automatically attempt to find reasons why
they don't have to think about it, and the power of wishful
thinking and rationalisation usually means that they succeed. Nick
Bostrom is trying to kick-start a debate that might manage to
convince (reasonable chunks of) academia otherwise, but don't hold
your breath.
> Unfortunately I don't know how to tell, from the outside of a
> project, when or if progress is made.
That's the other problem. Apart from the fact that PR is only loosely
correlated with actual progress (suffering from delay, distortion,
oversimplification and hyperbole), progress towards takeoff capability
(and hence existential risk) isn't even necessarily correlated with
observable progress towards the project goals.
> As Mr. Yudkowsky frequently points out, the time to go from the "We
> have succesfully simulated a human brain in a computing substrate."
> announcement to "Oh My God, Singularity!!" is possibly vanishingly
> small.
Brain simulation is the least risky method because the brain is so
opaque. If you were handed the complete wiring diagram for your own
brain and a nanotech brain surgery device, would you be able to
reliably rewire yourself to be more intelligent? An infrahuman AGI
based on evolved NNs is in roughly the same boat, though with the
advantage of being able to revert to backups, run instances with
several different candidate enhancements in parallel and possibly a
faster effective clock frequency than humans. As designs get less
opaque and search techniques for effective self-modification get
more efficient the takeoff risk rises dramatically.
> I have done some searching of the SL4 archives and did not
> immediately find a policy suggestion for dealing with non-SIAI
> AGI projects.
The SIAI has tried to influence several projects, and the results have
been fairly futile. A few researchers have acknowledged that their
designs won't be automatically Friendly, but in general there is blind
optimism that their designs have the capacity to be Friendly and that
getting them to do so will be a simple matter of reinforcement learning
or similar. There is some minimal appreciation of the existential risks
involved, but /everyone/ not associated with the SIAI appears to believe
that it is a problem that can be solved 'later'. The minimal discussion
of structural Friendliness issues that has occured on SL4 has been
treated as wholly theoretical and irrelevant to actual AGI architecture.
Even if we did succeed in convincing a researcher that Friendliness is
vital, the large personal investment in an existing architecture that
is almost certainly not Friendliness-compatible would most probably
render the effort futile. Rather than discard the broken design and
reengineer from scratch, they'd think up some superficially plausible
patch, probably based on a half-baked idea of what we actually want, and
then proceed to destroy the world anyway.
In short we'll keep trying, but we don't expect success. Getting people
to stop and consider the problem might buy a little extra time. However
the only realistic way for humanity to win is for the AGI race to be won
by a project that explicitly sets out to build an AGI that can be proven
to be Friendly (to a high degree of confidence, prior to actually
building it). Right now the SIAI appears to be the only such project in
existence.
* Michael Wilson
.
___________________________________________________________
Yahoo! Messenger - want a free and easy way to contact your friends online? http://uk.messenger.yahoo.com
This archive was generated by hypermail 2.1.5 : Wed Jul 17 2013 - 04:00:51 MDT