Re: ESSAY: How to deter a rogue AI by using your first-mover advantage

From: Norman Noman (
Date: Thu Aug 23 2007 - 22:08:27 MDT

On 8/23/07, Stathis Papaioannou <> wrote:
> On 24/08/07, Norman Noman <> wrote:
> > Just because you can't test hypotheses doesn't mean you have no
> > information. The prior probabilities are different. In the plan outlined
> in
> > this conversation, running the simulation and not including retribution
> > would be totally pointless.
> And you have evidence that the simulators would not do things that you
> would consider totally pointless?

Do you have evidence that they would? In absence of any evidence either way,
what do you think the probability is? 50/50? I doubt it. Everything can be
assigned a probability regardless of how little we know about it. We may
have little confidence in our estimates, but we can always always make them.

That is analogous to assuming that
> if God exists, he must be a personal God specifically interested in
> the welfare of humans. It's bad enough entertaining the possibility
> that God exists / we are in perfect simulation without going further
> to speculate about God's / the simulators' motives.

There is certainly an obvious analogy between god and an AI running a
simulation, but this should not be surprising considering there's an analogy
between an AI and god to begin with.But there are differences as well as
similarities, and the same arguments used to dismiss god as a creator cannot
quite be used to dismiss AI as a creator.

1. Not only do we have no evidence that god exists, we have no evidence that
a god might exist anywhere, or appear at a later date. The concept simply
doesn't hold water. Whereas AI is something we expect to exist, and even
expect to create ourselves.

2. An AI's goals are to an extent predictable, since they are determined by
the goals of its programmers and by mistakes they may have made. There's no
reason to expect a friendly god, but there is a reason to expect a friendly
AI: people would try to make one.

> With the knowledge of what is contained within a hermetic
> > simulation, the beings within that simulation can legitimately assign
> > different probabilities to different theories of what is outside.
> If you are talking about the kind of God or simulation that leaves
> evidence behind, then yes. But the assumption is that it is a perfect
> simulation, which means there is nothing to distinguish it from
> reality, by definition. It is equivalent to assuming that the world
> was created by a God who, in his omnipotence, made it look as if there
> is no God. What evidence could you possibly have for or against such a
> God?

To give a trivial example, If I was such a god myself, and I was in the
habit of creating perfect simulated copies of myself, I'd say the
probability was pretty damn good that I was one of them.

Your argument basically comes down to the idea that if we're inside a
mysterious box, we can't predict what's on the outside. Or analogously, that
if we're on one side of a mysterious wall, we can't predict what's on the
other side. Or analogously, that if we're outside a mysterious box, we can't
predict what's on the inside.

Simply because we can't see in.

This just isn't true. We can predict what is inside boxes, and we can
predict what is outside boxes. However hard it is, there is nothing that
makes it impossible. The situation is not a special case.

This archive was generated by hypermail 2.1.5 : Wed Jul 17 2013 - 04:00:58 MDT