Re: ESSAY: How to deter a rogue AI by using your first-mover advantage

From: Stathis Papaioannou (
Date: Fri Aug 24 2007 - 01:08:27 MDT

On 24/08/07, Norman Noman <> wrote:

> > And you have evidence that the simulators would not do things that you
> > would consider totally pointless?
> Do you have evidence that they would? In absence of any evidence either way,
> what do you think the probability is? 50/50? I doubt it. Everything can be
> assigned a probability regardless of how little we know about it. We may
> have little confidence in our estimates, but we can always always make them.

There are some things you have zero chance of guessing, like a
particular member of the set of all integers.

> > That is analogous to assuming that
> > if God exists, he must be a personal God specifically interested in
> > the welfare of humans. It's bad enough entertaining the possibility
> > that God exists / we are in perfect simulation without going further
> > to speculate about God's / the simulators' motives.
> There is certainly an obvious analogy between god and an AI running a
> simulation, but this should not be surprising considering there's an analogy
> between an AI and god to begin with.But there are differences as well as
> similarities, and the same arguments used to dismiss god as a creator cannot
> quite be used to dismiss AI as a creator.
> 1. Not only do we have no evidence that god exists, we have no evidence that
> a god might exist anywhere, or appear at a later date. The concept simply
> doesn't hold water. Whereas AI is something we expect to exist, and even
> expect to create ourselves.

OK, but that just replaces the concept of God with the super-AI:
something like Tipler's Omega Point.

> 2. An AI's goals are to an extent predictable, since they are determined by
> the goals of its programmers and by mistakes they may have made. There's no
> reason to expect a friendly god, but there is a reason to expect a friendly
> AI: people would try to make one.

If you discount the argument from intelligent design for a real world,
why would you use it for a simulated world? That is, why assume that
we are in a simulation designed to produce beings in the simulators'
image rather than the accidental result of a program trying various
starting parameters that may or may not give rise to a universe?

> > > With the knowledge of what is contained within a hermetic
> > > simulation, the beings within that simulation can legitimately assign
> > > different probabilities to different theories of what is outside.
> >
> > If you are talking about the kind of God or simulation that leaves
> > evidence behind, then yes. But the assumption is that it is a perfect
> > simulation, which means there is nothing to distinguish it from
> > reality, by definition. It is equivalent to assuming that the world
> > was created by a God who, in his omnipotence, made it look as if there
> > is no God. What evidence could you possibly have for or against such a
> > God?
> To give a trivial example, If I was such a god myself, and I was in the
> habit of creating perfect simulated copies of myself, I'd say the
> probability was pretty damn good that I was one of them.

And would the probability that you were one of them suddenly drop if
you decided to stop creating the simulations?

> Your argument basically comes down to the idea that if we're inside a
> mysterious box, we can't predict what's on the outside. Or analogously, that
> if we're on one side of a mysterious wall, we can't predict what's on the
> other side. Or analogously, that if we're outside a mysterious box, we can't
> predict what's on the inside.
> Simply because we can't see in.
> This just isn't true. We can predict what is inside boxes, and we can
> predict what is outside boxes. However hard it is, there is nothing that
> makes it impossible. The situation is not a special case.

You can make some guess in the case of the box from its size, shape,
history etc. You can't predict what is inside a box drawn at random
from the set of all possible boxes.

Stathis Papaioannou

This archive was generated by hypermail 2.1.5 : Wed Jul 17 2013 - 04:00:58 MDT