**From:** Ben Goertzel (*ben@goertzel.org*)

**Date:** Fri Feb 24 2006 - 22:15:56 MST

**Next message:**Philip Goetz: "Re: ESSAY: Program length, Omega and Friendliness"**Previous message:**Mike Dougherty: "Re: ESSAY: Program length, Omega and Friendliness"**In reply to:**Peter de Blanc: "Re: Why playing it safe is the most dangerous thing"**Next in thread:**sam kayley: "Re: Why playing it safe is the most dangerous thing"**Messages sorted by:**[ date ] [ thread ] [ subject ] [ author ] [ attachment ]

Hi,

*> You're trying to solve a decision problem: whether to try for FAI or
*

*> rush for the Singularity.
*

Actually I am not trying to solve exactly that decision problem; I am

working on AGI and FAI at the same time and trying to understand the

relation between them....

At some point my colleagues and I may need to try hard to solve that

decision problem in a more rigid sense -- if I have a powerful AGI on

hand and I have to decide whether to set a switch that will let it

start self-modifying and hence potentially move toward hard takeoff.

I am not at that point now....

I have however decided to spend more of my time working on AGI design

and engineering than on FAI theory...

*> In order to solve a decision problem, you need
*

*> to apply a decision theory.
*

This is not necessarily true, actually...

*>The most powerful and general decision
*

*> theory we have is based on Bayesian Probability Theory.
*

Well, probability theory is very valuable when one has enough data to

feed into it. In the case of the particular decision you cite, I

don't think there is enough data that current probability-theory

methods are very helpful.

*> Maybe you think that BPT-based decision theory is imperfect, but your
*

*> Indifference-based decision theory is weaker and less powerful than
*

*> Bayesian decision theory.
*

I am not basing any of my own personal or professional decisions on an

"indifference-based decision theory," at least not consciously or

explicitly.

I cited the indifference principle very explicitly as part of a

(possibly misguided) attempt to clarify SOMEONE ELSE'S ARGUMENT made

on this list, not as part of a presentation of my own point of view.

I'm sorry that this led to misunderstanding. I believe I was quite

explicit in my wording, though.

*> If you're not confident enough in Bayesian
*

*> decision theory to solve this problem, then you need to develop a
*

*> stronger decision theory, not a weaker one!
*

In fact I have spent a fair bit of effort working on ways to augment

probability theory with heuristics that are effective in cases when

there is not enough data to apply current probabilistic methods. (In

principle I consider this a weak way out, because I have a feeling

that "purer" probabilistic methods probably do suffice for all cases

of pragmatic reasoning in the classical-physics domain: but

probability math is far from fully developed, and in practice there

are a lot of practical decision problems where there is not enough

data to make judgments using known probability methods.)

*> > As a side point: After all, the general applicability of classical
*

*> > probability theory is *already* in question even within the human
*

*> > world, via Youssef's quantum probability theory...
*

*>
*

*> When I talk about Probability Theory, I mean in the Bayesian sense. If
*

*> there's a conflict between QM and Bayesian Probability Theory, you'll
*

*> have to point it out to me.
*

The precise nature of the conflict depends on the QM formalism that you choose.

You can choose quantum logic + classical probability theory, or you

can choose Boolean logic + complex probability theory.

See the papers here

http://physics.bu.edu/~youssef/quantum/quantum_refs.html

for information on the latter option

Either approach formalizes the way in which the quantum world violates

standard probability theory.

-- Ben

**Next message:**Philip Goetz: "Re: ESSAY: Program length, Omega and Friendliness"**Previous message:**Mike Dougherty: "Re: ESSAY: Program length, Omega and Friendliness"**In reply to:**Peter de Blanc: "Re: Why playing it safe is the most dangerous thing"**Next in thread:**sam kayley: "Re: Why playing it safe is the most dangerous thing"**Messages sorted by:**[ date ] [ thread ] [ subject ] [ author ] [ attachment ]

*
This archive was generated by hypermail 2.1.5
: Wed Jul 17 2013 - 04:00:56 MDT
*