From: Wei Dai (weidai@weidai.com)
Date: Wed Jul 16 2008 - 19:30:10 MDT
Consider two (contemporary) corporations that want to cooperate with each
other for mutual gain. They have two general options, trade or merge, and
the choice can be seen as a tradeoff between different kinds of overhead
costs. For trading, these include bargaining, contract enforcement, and
missed opportunities due to asymmetric information. Merging avoids some of
these, but increases agency costs (see
http://en.wikipedia.org/wiki/Agency_cost).
It occurs to me that two AIs who want to cooperate with each other face the
same choices, but merging might be a much more attractive option for them.
In case it's not obvious, "merging" here means something like creating a
third AI that will try to optimize a weighted average of the two AI's
utility functions, and then transferring all information and physical assets
to this new AI. The reason that merging is more attractive is that trading
still retains the same transaction costs, but merging doesn't incurs
perpetual agency costs. Instead, each of the original AIs only need to
verify during initial construction of the new AI that it will in fact try to
optimize the agreed upon utility function. (This seems much easier than
proving or verifying what source code an existing AI is running.)
If this analysis is correct, it may be that any society of AIs will
voluntarily merge into a singleton, in order to maximize gains from
cooperation. This singleton will then try to maximize a weighted average of
all of the original AIs' utility functions.
What about other kinds of minds, for whom merging may not be possible? Well,
if they are capable of self-modification, a group of such minds can agree to
modify themselves to each maximize a common combined utility function, and
that should work just as well as merging, as long as the agreement can be
enforced and verified (say with the help of a trusted third party). This
seems to imply that the ability to self-modify will lead to voluntary
borgification, but this process may stop short of a singleton (because when
there are only two such "borgs" left, who will act as the trusted third
party?)
This line of argument may seem to depend on the assumption that
intelligence==optimization process, which is something I've previously
argued against [1]. However, it may be that an intelligence, once it
satisfies all of its goals that can not be modeled as an optimization
process (such as finding answers to the philosophical questions in [1]), the
working on remaining goals can be seen as an optimization process.
(For nit-picking decision theorists, please take "maximize combined utility
function" to mean "maximize expected utility under a linear combination of
the individual priors and a linear combination of the individual utility
functions.)
[1] http://www.nabble.com/answers-I%27d-like-from-an-SI-td14007499.html
This archive was generated by hypermail 2.1.5 : Wed Jul 17 2013 - 04:01:03 MDT