From: Sebastian Hagen (sebastian_hagen@gmx.de)
Date: Tue May 10 2005 - 01:57:59 MDT
Russell Wallace wrote:
> Sure, but that's only one failure mode. What if the extrapolated
> volition converges, but ends up leading to a region of state space
> that doesn't contain sentient life?
The last judge is more of a final check to protect against unexpected
failures; it shouldn't be necessary to avoid likely failure modes. Why
do you think that this specific one is sufficiently likely to warrant
specifically worrying about it?
Either 'sentient life' matters to humanity in general, in which case I'd
expect the Extrapolated Collective Volition of humanity to ensure that
it continues to exist (or fail harmlessly in case of nonconvergence,
computational intractability, etc), or it doesn't, in which case I don't
have any particular reason to care if it is optimized away.
The conventional assumption in this case is that unupgraded humans, even
with some predictive help from a RPOP, are not capable of reliably
making good decisions about rules that are supposed to stay in force
forever. This applies to the rules for individual domains, but it also
applies to the whole idea of domain protection.
If there are critical differences in the values of different subsets of
the population the CV still has the option implement an individual or
domain protection system with permanently fixed rules; it would be in a
much better position to make that decision than present-day humans.
Sebastian Hagen
This archive was generated by hypermail 2.1.5 : Wed Jul 17 2013 - 04:00:51 MDT