From: Stuart Armstrong (dragondreaming@googlemail.com)
Date: Sat Jun 28 2008 - 04:33:39 MDT
>> If the AI is designed to be ecstatically happy carrying out
>> the orders of the snail, and to be a tiny bit happy when being smart
>> and consistent, then (as long as these happiness scores are not
>> cumulative, and the AI does not get happy now because of potential
>> future happiness) it will cheerfully follow the snail forever. (the
>> happiness assumptions are what is needed if we are to say that one
>> goal is strictly preferred to another).
>
> Your caveat (below) is appreciated, but I do have a question
> here, probably because I didn't start following this thread until very
> lately.
> What about this objection to "If the AI is designed to be ecstatically
> happy..."? Don't we often assume here that the AI manages to get
> hold of its own source code? Doesn't this allow that, given time,
> the AI may deviate from its initial programming, i.e., that even though
> it's being guided as it does consider and rewrite, it still has the
> potential
> of sooner or later deviating from any initial characterization.
Instruct the AI to only rewrite it's own source code if doing so will
reinforce its higher goals. Though I admit my shorthand of "happy" was
sloppy; it implies that there might be a shortcut the AI would take,
that its goal is to be happy. But it's real goal is to serve the
snail.
Stuart
This archive was generated by hypermail 2.1.5 : Wed Jul 17 2013 - 04:01:03 MDT