Re: [agi] A difficulty with AI reflectivity

From: Eliezer Yudkowsky (
Date: Thu Oct 21 2004 - 13:19:06 MDT

Michael Roy Ames wrote:
> Eliezer: please give us your definition of the phrase "wrap-around
> reflectivity".

Schimdhuber's original claim, that the Godel Machine could rewrite every
part of its own code including tossing out the need for a theorem-prover, I
would consider wrap-around reflectivity. A human's ability to have this
kind of email conversation - reflect on different possible architectures
for reflectivity - I would consider an even higher kind of wrap-around, and
the kind I'm most interested in. But I prefer to have somewhere to ground
the discussion, like reflective formal systems, otherwise people start
giving verbal answers.

> That is an adamant statement and one that I would agree with (except perhaps
> for the 'ever' part). The ability of humans to perform wrap-around
> reflectivity (as I understand it) is an illusiary one. We perform the same
> algorithmic techniques when working through proofs that a computer would
> use, but fall back on heuristics quickly when we detect our own tower of
> meta becoming absurd.

We are algorithms, but I don't think we're doing the same sort of thing
that a reflective theorem-prover would do. For example, humans can
actually be temporarily confused by "This statement is false", rather than
using a language which refuses to form the sentence. We don't run on
towers of meta. We're doing something else.

Eliezer S. Yudkowsky                
Research Fellow, Singularity Institute for Artificial Intelligence

This archive was generated by hypermail 2.1.5 : Wed Jul 17 2013 - 04:00:49 MDT