From: Eliezer S. Yudkowsky (sentience@pobox.com)
Date: Tue May 09 2006 - 17:29:51 MDT
Mikko Särelä wrote:
>
> My guess is that he is stating that you cannot prove any possible self
> modification to either to be conclusively good or bad. Where good or bad
> is judged by any reasonable criterion. Which means that you either have to
> narrow your self modifications to those that can be proven good, or to
> make some sort of guesses and jump to a possible unknown.
That you can't prove this for every possible self-modification is
obvious by Rice's Theorem, an extension of the halting problem which
says that if a property of a program's output is nontrivial, it is not
possible to say in finite time whether any possible program has that
output. This includes things like simple multiplication.
Nonetheless human engineers build CPUs that carry out multiplication, by
working in the restricted space of CPU designs that the chip engineer
*knows* carries out multiplication, which is a subset of all possible
chip designs that carry out multiplication. To build a bridge that
stays up, it is not necessary to be able to calculate of any possible
bridge whether it stays up. It is enough to know at least one bridge
design such that you can calculate that it supports at least 30 tons.
-- Eliezer S. Yudkowsky http://intelligence.org/ Research Fellow, Singularity Institute for Artificial Intelligence
This archive was generated by hypermail 2.1.5 : Wed Jul 17 2013 - 04:00:56 MDT