**From:** Ben Goertzel (*ben@goertzel.org*)

**Date:** Wed Feb 22 2006 - 06:10:36 MST

**Next message:**Richard Loosemore: "Re: ESSAY: Program length, Omega and Friendliness"**Previous message:**turin: "Re: Think of it as AGI suiciding, not boxing"**In reply to:**Jeff Medina: "Re: ESSAY: Program length, Omega and Friendliness"**Next in thread:**Richard Loosemore: "Re: ESSAY: Program length, Omega and Friendliness"**Reply:**Richard Loosemore: "Re: ESSAY: Program length, Omega and Friendliness"**Reply:**Eliezer S. Yudkowsky: "Re: ESSAY: Program length, Omega and Friendliness"**Messages sorted by:**[ date ] [ thread ] [ subject ] [ author ] [ attachment ]

On 2/22/06, Jeff Medina <analyticphilosophy@gmail.com> wrote:

*> William Pearson wrote:
*

*> > So the maximum number of bits of Phi and hence number of known or
*

*> > provably friendly programs is bounded by the length of your starting
*

*> > program.
*

*>
*

*> As Eliezer mentioned, the number is infinite, and hence unbounded
*

It seems more interesting to me to observe that the maximum

algorithmic information of any program in the set of provably friendly

programs is finite, and is bounded by the length of your starting

program. (here "provably" is taken to mean "provably by your starting

program")

(The algorithmic information of X is defined, loosely, as the length

of the shortest program for computing X)

On the other hand, if the starting program is allowed to increase its

algorithmic information over time via adding new physical compute

resources, then things get more complicated -- but this doesn't get

around the basic problem and obvious I've cited above.

Chaitin has summarized Godel's theorem nicely as "You can't use a ten

pound formal system to prove a twenty pound theorem."

The application of this to Friendly AI would seem to be: "A ten pound

intelligence can't prove the Friendliness of a twenty pound

intelligence."

Furthermore, it seems to me that "A ten pound intelligence can't prove

any solid probabilistic bounds on the Friendliness of a twenty pound

intelligence" either....

This suggests that *perhaps* mathematical proof is not the right

paradigm for us to be using to think about these issues... (Remember,

"mathematical proof" is just an abstract construct found useful by us

humans in particular situations we have encountered so far...)

-- Ben

**Next message:**Richard Loosemore: "Re: ESSAY: Program length, Omega and Friendliness"**Previous message:**turin: "Re: Think of it as AGI suiciding, not boxing"**In reply to:**Jeff Medina: "Re: ESSAY: Program length, Omega and Friendliness"**Next in thread:**Richard Loosemore: "Re: ESSAY: Program length, Omega and Friendliness"**Reply:**Richard Loosemore: "Re: ESSAY: Program length, Omega and Friendliness"**Reply:**Eliezer S. Yudkowsky: "Re: ESSAY: Program length, Omega and Friendliness"**Messages sorted by:**[ date ] [ thread ] [ subject ] [ author ] [ attachment ]

*
This archive was generated by hypermail 2.1.5
: Wed Jul 17 2013 - 04:00:55 MDT
*