From: Philip Goetz (philgoetz@gmail.com)
Date: Mon Nov 20 2006 - 13:44:58 MST
On 11/13/06, Eliezer S. Yudkowsky <sentience@pobox.com> wrote:
> Joshua Fox wrote:
> > Here's a question I have been wondering about. I'm sure that it has been
> > addressed somewhere. If someone can point me to a URL, I'd appreciate it.
> >
> > If multiple near-AGIs emerge, then basic Darwinian arguments show that
> > the one that reproduces itself the best will have the most copies; and
> > mutations favoring survival will spread.
>
> See the thread "Darwinian dynamics unlikely to apply to
> superintelligence" in January 2004:
>
> http://www.sl4.org/archive/0401/index.html#7506
I read it, but I didn't understand it. I could not find any logical
implications in it. A central paragraph is,
"It follows that we have no reason to expect any SI we deal with to attach
a huge intrinsic utility to its own survival. Why? Because that's an
extremely specific outcome within a very large class of outcomes where the
SI doesn't shut itself down immediately. There is, in other words, no
Bayesian evidence - no likelihood ratio - that says we are probably
looking at an SI that attaches a huge intrinsic utility to its own
survival; both hypotheses produce the same prediction for observed behavior."
There doesn't seem to be any reason given for the conclusion. I could
just as well say, "We have no reason to expect any animal we deal with
to attach utility to its own survival, because that's an extremely
specific outcome within a very large class of animals who don't commit
suicide immediately."
This archive was generated by hypermail 2.1.5 : Wed Jul 17 2013 - 04:00:57 MDT