From: Peter C. McCluskey (pcm@rahul.net)
Date: Wed Jun 07 2006 - 16:05:57 MDT
Your AI Risks paper is a good deal better than your previous papers.
Your argument for a hard takeoff is unconvincing, but at least it looks
like a serious attempt at an argument (in contrast to your prior assertions
which looked like pure handwaving).
Your nuclear fission analogy seems to provide a strong hint as to why
you expect a faster takeoff than I do.
I doubt that that analogy is appropriate because it describes a regime
change from exponential decrease to one of exponential increase. Whereas
with intelligence we've had exponential increase for quite some time.
We've even had some changes in the techniques used to drive the increase
in intelligence. We probably had a switch from random selection in most
apes to sexual selection that used proto-human brains to guide breeding
efforts. Then we developed cultural aids that enable some of our abilities
to improve in a Lamarckian fashion. These produced accelerations that were
too fast for bonobos and neanderthals to cope with, but not too fast that
they couldn't have observed and reacted to it if they had had the wisdom to
evaluate the risks and the resources to respond with.
Do you have an argument for why the fission analogy is likely to be more
apt than past changes in the way that increases in intelligence were driven?
On page 22, you say:
No system is secure that searches for ways to defeat its own security.
If the AI would harm humanity in any context, you must be doing
something wrong on a very deep level, laying your foundations awry.
I get the impression that you think these two sentences say roughly the
same thing, but I agree with the first but find the second overly broad
in a way that you are still biasing your description of the problem toward
the a solution similar to the one you thought you had in CFAI. The risk
that an AI would want to harm humanity if its creator told it to do so is
not the same kind of risk as the risk of an AI being designed with a goal
that would imply harming humanity. You don't appear to have presented an
argument why it is essential to solve the broader set of risks in the AI
software.
On page 27, you sound unreasonably confident of an AI's ability to
build nanotech devices. AI access to a sophisticated quantum computer
would probably be sufficient for the approach you describe to work on
the first attempt. But with more conservative assumptions about the
cpu power available to it, everything I know about molecular modelling
suggests that the AI would not be able to produce good enough models
to produce a working device on it's first few attempts.
Your argument is not very sensitive to the probability that this scenario
will work the first time. But the way you phrase your argument suggests
that you tend to seriously overestimate your ability to forecast big
innovations and are biased toward underestimating their difficulty,
which reduces my confidence in your ability to analyze AI risks.
On page 29 you use the word majoritarian in ways that seem confusing.
Are you switching between referring to a majority of humans and a majority
of AIs? If so, you need to be a little clearer that you're making that
switch.
On page 33 you say: "Furthermore, to scan a human brain neuron by neuron,
we need sophisticated nanotechnology." That seems inconsistent with
Merkle's analysis of how it could be done with technology available in
1989 in this paper: http://www.merkle.com/merkleDir/brainAnalysis.html.
The difficulty of understanding human minds well enough to improve them
rapidly seems to be the main problem with the human enhancement scenario,
so the effort need for uploading seems unimportant to your arguments, and
you could remove that paragraph without affecting the rest of the paper.
-- ------------------------------------------------------------------------------ Peter McCluskey | Science is the belief in the ignorance of experts. www.bayesianinvestor.com| - Richard Feynman
This archive was generated by hypermail 2.1.5 : Wed Jul 17 2013 - 04:00:56 MDT