From: Michael Vassar (michaelvassar@hotmail.com)
Date: Tue Aug 30 2005 - 14:43:49 MDT
> > Three discrete top-level goals might easily interact in such a
> > manner as to alter or remove one-another, but they should not
> > ever generate novel top-level goals, only sub-goals.
>
>That's a design decision. We have already supposed that the
>organism has write access to its top-level goals, thus
>violating that design decision.
I don't think so. The top level goals *can* write new top level goals (you
really couldn't prevent an AI from doing this by denying itself access to
its top level goals. If you tried it would just write a new AI with
different goals and delete itself), but it will only do so if the expected
utility of instituting a new goal as top level is greater than that of
instituting it as a sub-goal *from the perspective of its current top level
goal system*. In a human, this might indeed be the case. Our goals work
better when they are independently supported. Lower level goals suffer from
attenuated affect, etc. As a result, it frequently serves our top level
goals to add new top level goals. There is no reason to design an AI which
suffers from our difficulties in the implementation of goal heirarchies
however, so why would it implement new supergoals? Seriously, it seems to
me that you are saying that you can see a way in which its behavior could
accidentally lead to low utility outcomes, yet if that is the case, why
don't you expect it to see that same potential outcome and avoid it, at
least once it is human equivalent. At any given time, a FAI will be acting
to maximize its utility function. It is possible that in some cases,
changing supergoals would maximize its current utility function, and in
those cases it would do so, but those cases are specifically not something
we want to avoid. We want it to maximize its utility function, so long as
the utility function in question is Friendly.
> > In so far as the meme complex is a semantic
> > web, it cannot even directly interface with the perceptions
> > which it is referring to,
>
>This statement combines two large assumptions - that the meme
>complex "is" a semantic web, and that a semantic network can't
>"directly" interface with perceptions. My belief, as expressed
>in my 2000 Cognitive Science paper "A neuronal basis for the
>fan effect", is that the nodes in semantic networks can and
>probably should actually be recalled, content-addressable memories
>stored in networks. One of the reasons for doing this is so
>that they can interface with perceptions. (In fact, the inability
>of symbolic AI to interface with perception is one of the main
>motivating factors for adding a subsymbolic level.)
I was aware that I was making large assumptions about how human minds
actually work, at a very crude but useful approximation, hence "in so far
as". My statement, like yours, can be taken to be plausible speculations,
or if you find it implausible I will be happy to hear why in detail off
list. The argument is not dependent on the assumptions in question.
>The use of the term "friendliness" has fooled many of us into
>thinking that we know what we're talking about, when others on
>the list apparently have a technical definition in mind.
I am afraid that this is the case. At any rate, the document CAFAI is not
very long.
It would probably be helpful to everyone if we consistantly used the phrase
Powerful Optimization Process Encompassing Human Moral Complexity POPEHMC or
some more euphonious acronym, rather than FAI. It's probably too late to
conveniently change names to
"The Singularity Institute For Optimal Processes", "The Singularity
Institute for Compatible Preferences", or anything of the sort, but it might
not be. The phrase AI has LOTS of emotional baggage and is associated with
all sorts of inappropriate associations. The phrase for Artifical
Intelligence is also somewhat misleading, as we are only *for* certain types
of Artificial Intelligence.
This archive was generated by hypermail 2.1.5 : Wed Jul 17 2013 - 04:00:52 MDT