From: Scott Yokim (firstname.lastname@example.org)
Date: Mon Jul 24 2006 - 14:44:29 MDT
In February, Eliezer said:
<quote>Reflective decision theory - a theory of motivationally stable
self-enhancement - is the world's second most important math problem.
The *most* important math problem is how to phrase the motivational
invariant itself. A classical utility function *probably* isn't going
to cut it. My suspicion is that being able to build a reflective
decision system, I would know a great deal more about my options for
motivational invariants, and how to structurally describe those
structurally complex things that humans want - such as "free will" or
"freedom from having one's life path too heavily optimized by outside
sources as opposed to one's own efforts". I am doubtful I can solve the
most important math problem without having solved the second most
important math problem first. Sadly and dangerously, FAI knowledge
*always* lags behind AGI knowledge because AGI is a strictly simpler
Not only does one need to (learn how to) specify goal preservation in decision theory, but also to learn how to preserve more than one (conflicting!) goal at a time (do no harm to the human race, fulfill the universe's destiny, etc.).
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around
This archive was generated by hypermail 2.1.5 : Mon May 20 2013 - 04:01:04 MDT