From: Tim Freeman (tim@fungible.com)
Date: Sun Apr 13 2008 - 08:26:38 MDT
On Sat, Apr 12, 2008 at 10:00 PM, Vladimir Nesov <robotact@gmail.com> wrote:
> But how do we figure it out?
From: "Nick Tarleton" <nickptar@gmail.com>
>Well, like he said, it's *what you care about*, which is not arbitrary
>(or, rather, what you currently care about is contingent, but the fact
>that you already have values means you can't pick any random
>extrapolation), and which people have reasoned about for quite a
>while.
What's the purpose of this speculation? I can imagine several possibilities:
A. Be ready in case we get to the point where we can self-modify more
effectively than now and replace our present unconscious utility function
with something we figured out. In that case, the utility function
should describe human desires. But what's the point in this
proposed self-modification? We might figure it out wrong, and
the copy can't be more accurate than the original, so I see risk
without benefit.
B. Be ready in case we can build something new that has a utility
function we can specify. In that case, the utility function should
describe desires that are compatible with humans liking the actions
chosen by this new something. No human really has the satisfaction
of other humans as their highest value, so in this case the problem
is to design a new utility function. If this is the plan, I'm
confused by Eliezer saying that the utility function is not
up for grabs.
C. One-upmanship, to win an argument with a bunch of people you met on
this mailing list. In that case, you want to prove the other guy wrong,
so if he says A you want to imply that A is stupid and B is
obviously the goal, and vice versa, and of course C isn't the goal
at all because one-upmanship is bad.
-- Tim Freeman http://www.fungible.com tim@fungible.com
This archive was generated by hypermail 2.1.5 : Wed Jul 17 2013 - 04:01:02 MDT