From: Jef Allbright (jef@jefallbright.net)
Date: Sat Nov 10 2007 - 10:13:34 MST
On 11/10/07, Wei Dai <weidai@weidai.com> wrote:
> One of the things I'd like most from a friendly SI is answers to certain
> questions. If I meet an SI and it can't answer these questions or at least
> explain why they can't be answered, I would be really disappointed.
I expect that you would be disappointed and frustrated as the higher
intelligence would appear to respond in riddles. Pressed further, it
might respond [cf. Eliezer] that there are no mysterious answers, but
only mysterious questions, and that when you have composed a proper
question you will have the answer you seek.
Note that we can expect a more advanced intelligence to demonstrate
objectively superior results (relative to a given goal) but this does
not imply that it could effectively explain it to a lesser
intelligence.
Some alternatives:
It could provide the **appearance** of a satisfactory answer.
It could suggest the direction of a path leading to the understanding you seek.
It could manipulate you in the direction of a path leading to the
understanding you seek.
It could divert your attention toward other interests that you might
might find more productive at your current stage of development.
It could divert your attention toward pursuits that it would assess as
more productive.
It could place you in a simulation where some annoying person on a
email list tries to impress upon you the importance of context...
And on and on.
- Jef
This archive was generated by hypermail 2.1.5 : Wed Jul 17 2013 - 04:01:00 MDT