From: Wei Dai (weidai@weidai.com)
Date: Mon Sep 15 2008 - 14:55:52 MDT
Matt Mahoney wrote:
> For example, a group of miners are trapped underground in an accident. It
> will take 48 hours to dig them out, but unfortunately there is only enough
> air for the group to survive for 24 hours. The individual preferences are
> that everyone lives, which results in everyone dying. However, the social
> preference is that half of the miners kill themselves immediately so that
> the other half will have enough air until they can be rescued.
Matt, in your example the individuals are not rational in the Bayesian sense
if they each prefer that everyone lives. That's because under Bayesian
rationality they are supposed to try to maximize expected utility, which is
utility multiplied by probability. Because the probability of everyone
living is very low, they shouldn't prefer that everyone lives. Since the
topic under discussion is what happens when a group of individually rational
AIs merge together, I don't think your example is really relevant.
This archive was generated by hypermail 2.1.5 : Wed Jul 17 2013 - 04:01:03 MDT