From: Eliezer S. Yudkowsky (sentience@pobox.com)
Date: Sun May 07 2006 - 10:26:07 MDT
Mikko Särelä wrote:
> Hi Eliezer,
>
> I just watched your talk about recursion, the most important math problem
> and you mentioned changing value system as one of those problems that you
> have not yet been able to formulate and solve. I have some ideas, though
> not rigorously in math form, which I hope might help. Or you might already
> know all of these, but here goes anyways. Two different approaches, first
> one tells you when you possibly should change your value/goal system and
> the second one tells you how that should be aproached.
The problem I haven't been able to solve is *rigorously* describing how
a classical Bayesian expected utility maximizer would make changes to
its own source code, including the source code that changes the code;
that is, there would be some set of code that modified itself.
Classical decision theory barfs on this due to an infinite recursion.
Right now, I'm trying to solve this problem for an agent where Hume's
Guillotine applies; that is, the utility function itself would not
contain a dependency on states of the world. That's because this
version of the problem is simpler.
But Hume's Guillotine is simply false with respect to humans, just as
expected utility maximization is false with respect to humans. Human
beings and civilizations change what they conceive to be their goals,
moralities, and principles, as they think new thoughts and encounter new
evidence about matters of simple fact. This is more complicated than
expected utility maximization with a constant utility function, but it
is not magic. A simplified version might describe a Bayesian expected
utility maximizer which acts as if a semi-opaque but investigatable box
contains a written description of a computation which, when computed,
will specify the maximizer's utility function.
This complication seems orthogonal to the infinite recursion problem,
which is why I'm leaving it out of my current work.
The distinction you make between ends and means is already represented
within classical expected utility.
See also:
http://sl4.org/wiki/CoherentExtrapolatedVolition
http://sl4.org/wiki/DialogueOnFriendliness
-- Eliezer S. Yudkowsky http://intelligence.org/ Research Fellow, Singularity Institute for Artificial Intelligence
This archive was generated by hypermail 2.1.5 : Wed Jul 17 2013 - 04:00:56 MDT