Re: OpenCog Concerns

From: Matt Mahoney (
Date: Mon Mar 03 2008 - 10:24:58 MST

--- Jeff Herrlich <> wrote:

> I also agree that there are significant risks with the open-source approach.
> I think that some of those risks can be partially reduced by having a
> well-resourced, Safe-AI team building a closed-source AGI alongside
> improvements to the OpenCog model (eg. Novamente). IOW, keep the good guys
> "on top of the code".

I don't buy it.

Friendliness has nothing to do with keeping AI out of the hands of "bad guys".
Nobody, good or bad, has yet solved the friendliness problem.

Instances of OpenCog and Novamente would likely be peers in a distributed
query/message posting service like the one I proposed, or some other system
that manages the communication between billion of humans. Their success will
depend on how well and how fast they judge the quality and relevance of
information in a hostile, competitive environment.

As long as humans are the primary source of knowledge, the network will be
friendly because good service is a subgoal of the evolutionarily stable goal
of acquiring computing resources. I estimate a network will be deployed over
the next decade and it will remain friendly for about 30 more years. As the
machines do more of our thinking, humans will become less relevant and the
interchange between peers will evolve from natural language to something
incomprehensible and beyond our cognitive abilities to learn. Shortly
afterwards there will be a singularity.

You cannot ignore that there is a US $66 trillion per year incentive (the
value of all human labor worldwide) to develop distributed AI. I have seen
many proposals to build prototype friendly AI on a desert island, or otherwise
isolated or tightly controlled by its developers. I hope you can see how
impractical these approaches are. You can't compete with the computing power,
knowledge base, and user base already available on the internet.

-- Matt Mahoney,

This archive was generated by hypermail 2.1.5 : Wed Jul 17 2013 - 04:01:02 MDT