From: Eliezer Yudkowsky (sentience@pobox.com)
Date: Tue Jun 15 2004 - 17:31:28 MDT
There have been - not surprisingly, and it's a legitimate topic - many
recent questions of the form, "What if my volition isn't what I want?"
I would like to point out that, if your volition is not what you want,
there is a *reason* for it - it is not something that happens because of a
random-number generator.
The reason may be, "That idiot Eliezer screwed up the extrapolation
dynamic." If so, you got me, there's no defense against that. I'll try
not to do it.
The reason may also be, "My short-range volition is muddled and I possess a
medium-range volition coherent with that of the rest of humankind."
I'd like to take a moment to discourse on the importance of attaching
concrete meanings to abstractions. People get lost when they don't do
this. I've seen people try to manipulate math that they don't have a good
grasp for, use equations that they have memorized but not seen as obvious,
manipulate words piled atop words without a strong sense of what the words
mean. The same caution holds true for manipulating thoughts about
"volition" or "collective volition". I know what the words mean because I
invented them to describe specific things that I wanted to do for specific
reasons. I did try to list some of the reasons in "Collective Volition",
but I may have failed to convey a proper grounding. In fact, it seems
nearly certain that I have conveyed only a fraction of the grounding. I do
not make my choices at random; if you can't see why I would possibly want
to do something, you probably mis-visualized the thing I want to do.
So when I say: 'The reason may also be, "My short-range volition is
muddled and I possess a medium-range volition coherent with that of the
rest of humankind."'
I mean: 'The reason may also be, "I am making a decision based on really
stupid and messed-up reasons, and if I knew more and thought longer I'd
come to pretty much the same conclusion as everyone else who thinks about
the subject, and it wouldn't be the same as my current decision."'
Now, remember, I'm not saying that our extrapolated volitions should always
override our current decisions. I'm saying that my current decision is
that the decision as to whether our extrapolated volitions should override
our current decisions should be made by our extrapolated volitions. What
if your current decision disagrees with the decision of our extrapolated
collective volition about whether the decision should be made by your
current decision or your extrapolated volition? This means that:
1) Eliezer screwed up the dynamic.
2) Your current decision doesn't have a good handle on reality; there are
enormous consequences you don't foresee, and if you knew the true
consequences, you would repudiate your decision.
3) A nicer person who was otherwise extremely similar to you would
repudiate your decision.
4) Everyone else's volition doesn't agree with your decision, and there's
no way for you to convince Eliezer to let you personally take over the
world, and Eliezer isn't willing to personally take on the onus of writing
an individual volition dynamic because there's no way for a humane
superintelligence to veto his decision in case it turns out to be wrong.
There are all kinds of reasons why our volitions might contradict our
decisions, and most of them involve either being stupid - unforeseen
consequences, missing obvious solutions - or being not the people we wished
we were, i.e., the sort of reason Samantha is justly worried about being
extrapolated just the way she is rather than a wiser version of herself,
and that Robin doesn't want murderous thoughts to kill people.
It is noteworthy that criticisms seem to be equally divided between people who:
A) Think that humans are too awful for their volitions to be extrapolated.
(What am I supposed to extrapolate instead?)
B) Think that people's present-day decisions are just fine, and this whole
volition-extrapolating thing is unnecessary. (Are you absolutely sure
about that? You would go ahead and do it even if you knew that with
another few years to think about the subject, you would change your mind
and be horrified at your previous decision?)
Maybe I should let the two sides fight it out on their own, and then take
on the winner, if they haven't already struck a compromise that looks just
like "Collective Volition".
-- Eliezer S. Yudkowsky http://intelligence.org/ Research Fellow, Singularity Institute for Artificial Intelligence
This archive was generated by hypermail 2.1.5 : Wed Jul 17 2013 - 04:00:47 MDT