From: Keith Henson (hkhenson@rogers.com)
Date: Sun Jun 13 2004 - 20:25:42 MDT
At 05:58 PM 13/06/04 -0400, you wrote:
>Samantha Atkins wrote:
>>On Jun 13, 2004, at 7:56 AM, Eliezer Yudkowsky wrote:
>>>
>>>Samantha, you write that you might have a badly warped view of what kind
>>>of person you would like to be. "Badly warped" by what criterion that I
>>>feed to the FAI? Your criterion? Someone else's? Where am I supposed
>>>to get this information, if not, somehow, from you? When you write down
>>>exactly how the information is supposed to get from point A (you) to
>>>point B (the FAI), and what the FAI does with the information once it's
>>>there, you'll have something that looks like - surprise! - a
>>>volition-extrapolating dynamic. It's not a coincidence. That's where
>>>the idea of a volition-extrapolating dynamic *originally comes from*.
>>That is my point. The information is not necessarily available from the
>>person[s] in sufficient quality to make wise decisions that actually work
>>for the good of humanity.
>
>Okay... how do you know this?
I think the point is that she doesn't know for sure. ". . . not
necessarily . . ." I can see two sources for lack of knowledge. I don't
think we can ever predict something like individual radioactive decay. But
even if something is completely deterministic, we might not know all the
influences that shape the outcome.
And furthermore, changing environmental conditions make last week's "wise
decisions" less than wise. Consider how things would change if we found we
were going to get smacked in a few years by a dinosaur killer comet.
>Also, where do I get the information? Like, the judgment criterion for
>"wise decisions" or "good of humanity". Please note that I mean that as a
>serious question, not a rhetorical one. You're getting the information
>from somewhere, and it exists in your brain; there must be a way for me to
>suck it out of your skull.
Not when it isn't there.
Further, the question is poorly framed. "good of humanity" for
example. What is the more important aspect of
humanity? Genes? Memes? Individuals built by genes who are running
memes? I have been thinking around the edges of these problems for close
to two decades and I can assure you that I don't have *the* answer, or even
*an* answer that satisfies me. (Right now, of course, they are all important.)
That said, Eliezer is absolutely right to grope in this direction.
snip
Keith HEnson
This archive was generated by hypermail 2.1.5 : Wed Jul 17 2013 - 04:00:47 MDT