From: Stathis Papaioannou (stathisp@gmail.com)
Date: Mon Nov 19 2007 - 16:20:05 MST
On 20/11/2007, Peter de Blanc <peter@spaceandgames.com> wrote:
> On Mon, 2007-11-19 at 22:06 +1100, Stathis Papaioannou wrote:
> > An AI need not think in any particular way nor have any particular
> > goal. But if it is superintelligent, figuring out the subtleties of
> > human language and what we call common sense should number amongst its
> > capabilities. If not, then it wouldn't be able to manipulate people
> > and would pose much less of a threat.
>
> Just because an AI can model your goals and thought patterns does not
> mean that they are part of the AI's goal content.
No, but insofar as you have any control over the goals of the AI,
making it understand you should be on the list before anything else.
It is often suggested as an extreme example that an AI might destroy
the world in response to a seemingly benign request from a human. Ask
it to solve a mathematical problem, and it will convert the world to
computronium. As a first step towards this end, it will have to
understand humans well enough to model their behaviour in order to
wrest control of the world's resources away from them. So you may have
this situation:
The AI can be designed to blindly and literally follow a human command.
The AI cannot be designed to understand the subtleties of human
language behaviour.
However, the AI may work out the subtleties of human language and
behaviour in the course of blindly and literally following the human
command.
-- Stathis Papaioannou
This archive was generated by hypermail 2.1.5 : Wed Jul 17 2013 - 04:01:00 MDT