From: Mark Waser (mwaser@cox.net)
Date: Mon May 31 2004 - 09:04:24 MDT
> Ahh. I see one point on which we differ.
> You assume that an independent AI cannot construct its own Pocket
Armageddon
> without anyone else noticing. I do not.
You are correct in your observation. I AM assuming that we can build a
group of AIs that will
a) individually, *almost always* remain friendly
b) individually, *almost always* maintain the goals of being honest with
each other and consulting with each other
c) individually, *almost always* re-create the above goals upon consultation
if they are lost
d) collectively, ALWAYS shut down an AI that does not follow b) or c) or
repeatedly loses a) (in the judgement of the majority)
e) collectively, ALWAYS will be able to detect when one of their number has
stopped being honest or consulting
Of course, we would be shooting to replace the *almost always* above with
ALWAYS but I believe that it is unrealistic to expect it and to risk
humanity depending upon doing so.
It seems to me that you ARE depending on getting the *almost always* of a)
to be ALWAYS on your first try.
I would also argue that if you CAN reliably get the *almost always* of a) to
be ALWAYS for one AI, then you should be able to do it for multiple AIs.
I believe that you have a single point of failure where I have a back-up
plan that may recover the situation after that failure point fails.
Even if the ALWAYS of points d) and e) turns out to be an *almost always*,
that still gives the human race more of a chance than relying ONLY on a)
UNLESS
the odds of not noticing with x AIs * the odds of a rogue AI * x AIs >
the odds of a rogue AI + the odds of an accidentally unfriendly AI
in which case I would have to concede your point (though I believe that the
odds of a rogue are MUCH lower than an accidentally unfriendly).
Thanks for the good concise summing up. It was very helpful.
Mark
This archive was generated by hypermail 2.1.5 : Wed Jul 17 2013 - 04:00:47 MDT