Purpose of the speculation? (was Re: Bounded population)

From: Tim Freeman (tim@fungible.com)
Date: Sun Apr 13 2008 - 08:26:38 MDT

On Sat, Apr 12, 2008 at 10:00 PM, Vladimir Nesov <robotact@gmail.com> wrote:
> But how do we figure it out?

From: "Nick Tarleton" <nickptar@gmail.com>
>Well, like he said, it's *what you care about*, which is not arbitrary
>(or, rather, what you currently care about is contingent, but the fact
>that you already have values means you can't pick any random
>extrapolation), and which people have reasoned about for quite a

What's the purpose of this speculation? I can imagine several possibilities:

A. Be ready in case we get to the point where we can self-modify more
   effectively than now and replace our present unconscious utility function
   with something we figured out. In that case, the utility function
   should describe human desires. But what's the point in this
   proposed self-modification? We might figure it out wrong, and
   the copy can't be more accurate than the original, so I see risk
   without benefit.

B. Be ready in case we can build something new that has a utility
   function we can specify. In that case, the utility function should
   describe desires that are compatible with humans liking the actions
   chosen by this new something. No human really has the satisfaction
   of other humans as their highest value, so in this case the problem
   is to design a new utility function. If this is the plan, I'm
   confused by Eliezer saying that the utility function is not
   up for grabs.

C. One-upmanship, to win an argument with a bunch of people you met on
   this mailing list. In that case, you want to prove the other guy wrong,
   so if he says A you want to imply that A is stupid and B is
   obviously the goal, and vice versa, and of course C isn't the goal
   at all because one-upmanship is bad.

Tim Freeman               http://www.fungible.com           tim@fungible.com

This archive was generated by hypermail 2.1.5 : Wed Jul 17 2013 - 04:01:02 MDT