From: Eliezer S. Yudkowsky (sentience@pobox.com)
Date: Fri Nov 23 2007 - 00:31:51 MST
Norman Noman wrote:
>
> According to the poetry, "knew more" etc. is to be "interpreted as we
> wish that interpreted, extrapolated as we wish that extrapolated." DOES
> "knew more" happen first, interpreted in the explicit manner you
> describe, or is it "interpreted as we wish it interpreted", and if so,
> how is THAT interpreted?
>
> It would be really nice if you could break down the order of operations,
> and define them more explicitly.
You couldn't possibly begin to extrapolate volitions as people "would
wish" them extrapolated, without first doing a huge amount of
extrapolation to get the "would wish" part. If you can't write
something the complexity of CEV today, we'll have to extrapolate you
to get it. So you wouldn't even begin to start on the "extrapolated
as we wish that extrapolated" until you'd evaluated a whole lot of
"knew more" and "thought faster".
It may make sense to assume that CEV writes a single program to
replace itself, using only the first-order extrapolation, and any
second-order extrapolation of how to extrapolate would happen in this
replacement program if it happened at all.
-- Eliezer S. Yudkowsky http://intelligence.org/ Research Fellow, Singularity Institute for Artificial Intelligence
This archive was generated by hypermail 2.1.5 : Wed Jul 17 2013 - 04:01:00 MDT