**From:** Matt Mahoney (*matmahoney@yahoo.com*)

**Date:** Mon Oct 06 2008 - 06:58:20 MDT

**Next message:**Mike Dougherty: "Re: [sl4] prove your source code"**Previous message:**Eschatoon Magic: "[sl4] Next Cosmic Engineers meeting in Second Life: Discussion of CTRL-ALT-R: REBAKE YOUR REALITY"**In reply to:**Rolf Nelson: "Re: [sl4] prove your source code"**Next in thread:**Mike Dougherty: "Re: [sl4] prove your source code"**Reply:**Mike Dougherty: "Re: [sl4] prove your source code"**Messages sorted by:**[ date ] [ thread ] [ subject ] [ author ] [ attachment ]

--- On Sun, 10/5/08, Rolf Nelson <rolf.h.d.nelson@gmail.com> wrote:

*> A belated response to an interesting thread:
*

*>
*

*> On Mon, Jul 14, 2008 at 3:14 PM, Wei Dai
*

*> <weidai@weidai.com> wrote:
*

*>
*

*> > Putting aside the issue of superrationality for now, I
*

*> wonder if anyone
*

*> > else finds it plausible that two dissimilar SIs can
*

*> know each other's source
*

*> > code. If we assume that they start out without such
*

*> knowledge, but each
*

*> > wants the other gain it, what can they do? Can one SI
*

*> prove to another what
*

*> > its source code is? Or, is there some other argument
*

*> for why SIs might know
*

*> > each other's source code (beyond "we
*

*> don't know what SIs might be capable
*

*> > of, so we can't rule it out")?
*

*>
*

*>
*

*> I don't have a certain answer. You have a prover agent
*

*> P who's trying to
*

*> prove to an observer agent O that P's source code has
*

*> goal G. To be more
*

*> precise, O and P each start controlling their own region of
*

*> space. P wants
*

*> to prove that P's region of space is controlled by an
*

*> agent that wants to
*

*> achieve G.
*

*>
*

*> Maybe O asks to send a set of probes into P's space
*

*> that will "sample"
*

*> random regions of P's spacetime and confirm that
*

*> they're occupied by mini
*

*> AGI's that are actively enforcing G. O would worry that
*

*> P might rewrite the
*

*> probes with "everything checks out great" data
*

*> before allowing the probes to
*

*> return to O, so O would have to somehow ensure that the
*

*> probes' integrity is
*

*> maintained while outside the space that O controls. Perhaps
*

*> the probes could
*

*> contain a secret key that self-destructs (is impossible to
*

*> read) if the
*

*> probe is tampered with; if a probe comes back without the
*

*> same secret key
*

*> then O would know P destroyed the probe and constructed a
*

*> fake copy loaded
*

*> with false observations. Not sure how you could create such
*

*> a probe; maybe
*

*> quantum no-cloning theorems could be of assistance in
*

*> constructing it.
*

*>
*

*> If there is a benefit to smuggling weaponry into your
*

*> rival's space, then P
*

*> might want to audit that the probes are harmless before
*

*> allowing them into
*

*> his space. That part seems easier; even if there isn't
*

*> a way to scan a probe
*

*> for weaponry without threatening its secret, probably you
*

*> could still use
*

*> fancy cryptographic tricks so that P can scan and blow up
*

*> probes #5, 6, and
*

*> 17, and then prove afterwards to O that P's decision to
*

*> blow up those
*

*> particular probes was pre-determined, and therefore that
*

*> those probes
*

*> weren't blown up because those particular probes
*

*> stumbled upon Something
*

*> They Weren't Supposed To See.
*

*>
*

*> Once O verifies that X percent of P's space is filled
*

*> with G-friendly
*

*> nano-AI's, then O probably won't worry that some
*

*> anti-G time-bomb is hiding
*

*> in some corner of P's space, since the rest of the
*

*> G-friendly mini-AI's
*

*> wouldn't *want* such a time bomb to continue existing
*

*> and would search for
*

*> and destroy such a time bomb if there was a chance of it
*

*> existing.
*

*>
*

*> -Rolf
*

I think you need to define more precisely what it means for an agent to "know" another's source code. If you mean that agent A can simulate agent's B program such that for any input x, B can predict A(x), then it is not possible for A and B to know each other's code. If A knows B, then K(A) > K(B), where K is Kolmogorov complexity. Therefore it is not possible for B to also know A, which would imply K(B) > K(A). The best you can do is a probabilistic model that allows for some chance of error in predicting either A(x) by B or B(x) by A.

By similar reasoning, an agent cannot predict its own actions with certainty, which would imply K(A) > K(A).

-- Matt Mahoney, matmahoney@yahoo.com

**Next message:**Mike Dougherty: "Re: [sl4] prove your source code"**Previous message:**Eschatoon Magic: "[sl4] Next Cosmic Engineers meeting in Second Life: Discussion of CTRL-ALT-R: REBAKE YOUR REALITY"**In reply to:**Rolf Nelson: "Re: [sl4] prove your source code"**Next in thread:**Mike Dougherty: "Re: [sl4] prove your source code"**Reply:**Mike Dougherty: "Re: [sl4] prove your source code"**Messages sorted by:**[ date ] [ thread ] [ subject ] [ author ] [ attachment ]

*
This archive was generated by hypermail 2.1.5
: Wed Jul 17 2013 - 04:01:03 MDT
*