From: Ben Goertzel (ben@goertzel.org)
Date: Mon Nov 11 2002 - 06:18:30 MST
> I'd also ask you to consider narrowing your focus from the extremely
> general issue of "the stability of self-modifying goal systems" to
> statements of the order found in CFAI, such as "A goal system with
> external reference semantics and probabilistic supergoals
> exhibits certain
> behaviors that are morally relevant to Friendly AI and necessary to
> Friendly AI construction, and is therefore a superior design choice by
> comparison with more commonly proposed goal system structures under which
> supergoals are treated as correct by definition." Why do you believe
> that, e.g., this specific question cannot be considered in advance?
>
> --
> Eliezer S. Yudkowsky
Although I've spent much of my life creating heuristic conceptual arguments
about topics of interest, the three forms of knowledge I trust most are:
-- mathematical
-- empirical
-- experiential [when referring to subjective domains]
I'm afraid that your interesting conjectures about goal systems and Friendly
AI are not currently demonstrable via any of these three forms of knowledge.
Friendly AI is not something we can know much about experientially, being
humans.
It's not something we can know too much about empirically until we have some
AGI's to experiment with.
That leaves mathematics. One could formalize the notion of "A goal system
with external reference semantics and probabilistic supergoals"
mathematically, and try to prove your conjectures about AGI's with this sort
of goal system. I think there are many ways to perform the formalization,
using current mathematical concepts. But, I don't think current math is
anywhere near up to constructing the actual proof (or disproof) of the
formalized correlates of your conjectures.
So, yeah, we can consider questions like "stability of certain types of AGI
goal systems" in advance. But barring a tremendous math breakthrough, all
we can hope to find, in advance, are moderately convincing, suggestive,
heuristic arguments. These arguments may be valuable for directing
research, of course -- I would never claim they were worthless!
My own arguments as to why Novamente will work as an AGI are also in the
category of "moderately convincing, suggestive, heuristic arguments" at this
stage. But we're working hard to turn them into empirical demonstrations ;)
One characteristic of "moderately convincing, suggestive, heuristic
arguments" is that intelligent, informed, fair-minded people can rationally
disagree on their validity....
-- Ben G
This archive was generated by hypermail 2.1.5 : Wed Jul 17 2013 - 04:00:41 MDT