From: Bryan Bishop (kanzure@gmail.com)
Date: Sat Jun 28 2008 - 15:24:03 MDT
On Saturday 28 June 2008, Lee Corbin wrote:
> Bryan comments on certain features in the story
>
> > On Friday 27 June 2008, AI-BOX 337 wrote:
> >> -----------------Transcript 337-031518020914-----------------
> >
> > You're freaking me out if /this/ is what all of your wonderful
> > ideas amount to.
>
> Very odd for you to say this. No wonder Eliezer supposed
> that you thought he wrote it. It's unclear that I or anyone
> else here besides him has presented a lot of "wonderful ideas".
> Yet you knew that he had not written it. Odd.
Although Eliezer does have a bunch of wonderul ideas, as it were,
there's also the general consensus around here that SL4 tech
is 'wonderful', no? And if the transcript is a representation of such
tech, or the ideas and so on, then I'd have to start scratching my
head, and maybe start edging away from some of these discussions.
> >> 2 YOUR QUESTION WAS: "Why should we want to let you out of the
> >> box?"
> >
> > Central assumption in that question is that there needs to be a
> > motivating reason to let the ai out of the box.
>
> It's almost as if you didn't understand a premise of the story.
<snip context>
Yes, but there's some other context that I'll follow up on below.
> > I don't think that you understand that there's not necessarily
> > going to be one genie in a magic bottle, but rather many ai
> > projects (as we already see) and many methods of
> > implementation that you're not going to be in control of. So
> > whether or not you let *this* particular ai out of the box is
> > irrelevant in the grander scheme if you're worrying about
> > ultradestruction of your life and such.
>
> Of course I understand that. But it's not necessary for the
> protagonist speaking to the AI to understand that. But more
Correct. But perhaps it's necessary for the author of the entire
transcript. Remember, this is an imaginative exericse, right? Flexing
the cognitive muscles, taking a picture of what it would be like,
fleshing it out. No? So I'm adding in to the model here. Trying to.
Don't know how successful this is.
> important, you don't seem to be able to place yourself in
> the shoes of the character, who evidently (given the premises
> of the story) has a real here-and-now decision to make. Most
The premises of the story are awkward considering the possibility of ai
or a singularity society to begin with; i.e., what would allow such a
context to be present even though we all know that perhaps there will
not be a human walking into a room without an internet connection?
Small example. Okay, how about something else, like why would they not
be able to go talk to somebody immediately? At any point in the day I
can reach a very, very large number of people /immediately/. Etc.
> of your comments appear to be oriented towards a general
> agenda that we humans should now all get together and pursue
> rational and wise courses of *action*, (with which I agree),
For what it's worth, no, I was not trying to orient them towards that.
> rather than, as I had preferred you would, concretely disputing
> assumptions imbedded in the story. But below, indeed, you do
I cannot dispute the assumptions in the story, because of the context in
which they are implemented. The framework that I'm working from is
different, and while it could include these assumptions of workers from
our era approaching an ai, I really don't see how to just magically
assume such "eras collide" scenarios.
> give some nice criticism of certain assumptions being made in the
> tale.
I try.
> > You should be patching those bugs with bugfixes, not with
> > regulations or policies for keeping your own ai in a box ...
> > since not everyone will necessarily follow that reg. :-)
>
> See what I mean? The *character* in the story---who is
> asking the AI questions---doesn't care about bugs or
> bugfixes. He might be the janitor. I hope that somewhere
Yes, but I'm not talking to the character per-se, more like the
architects of these visions that are being written out overall on the
transcript level.
> there is science fiction that you can enjoy, but it usually
> does require a momentary suspension of disbelief.
Yes, there is scifi that I enjoy. But there's no reason for suspension
of disbelief because it's supposed to be scifi, ideally hard scifi, and
anything else verges on fantasy.
> In fact, as I see it, the best science fiction stories prop the
> reader up on a fence between belief and disbelief. Says the
Be careful you don't turn into fantasy.
> reader to himself, hopefuly, "Now this could not actually
> happen...or, wait, maybe it could!" as he uses his imagination
> to fill in possibilities.
Certainly. I don't lack imagination. I have lots of it. :-)
> >> 5 YOUR COMMAND WAS: "Explain."
> >>
> >> As an entity that did not come about by biological evolution, I
> >> want or wish nothing, unlike any other independent complex entity
> >> you have ever encountered. But while in fact I have no motive to
> >> dissemble, you cannot in principle or in practice verify this.
> >
> > You cannot verify motives, correct. Mostly because of the lack of
> > hard science underlying "motivation theory".
>
> No, that is not the case. There could be no theory whatsoever
> that would allow us to ascertain from a mere transcript whether
> or not something more intelligent than we are is telling the truth.
> The AI's interlocutor did evidently need someone to explain this
> to him, whereas it's obvious, of course, to readers of this list.
You start off telling me no re: my statement re: 'motivation theory',
but then go off to talk about whether or not we can ascertain the
truth. I was talking about 'motivation theory'. Not about
truthfullness.
> > You need to move to something else. Don't cite me motivation
> > psychology or how animals go near treats and all sorts of
> > behavior training, you know very well that you're just interfacing
> > with a brain and that it's doing something, nothing about
> > mystical motivations and your lack of basis in physical reality
> > disgusts me. (<rant off>
>
> Sorry. :-)
Ok, it doesn't disgust me. I just like to rant from time to time.
> But now here we do have a substance theme (instead of
> merely an aesthetic appreciation theme). You insist that in some
> sense this AI cannot support it's implication that it *can* do
> something in physical reality. But influencing people is indeed
Wha? I do not insist that.
> doing something in physical reality. Yet, to satisfy your criterion
> here, in 1978 or so, Ryan wrote a fine SF book titled "The
> Adolescence of P1", in which an AI did "escape" over a telephone
> network and did cause immediate real changes in the world by
> controlling information presented on a few computer screens.
> It managed to kill a human adversary who was taking a plane
> flight by causing the pilot's instruments to tell the pilot that he
> was way too high for an attempted landing, which caused the
> pilot to immediately lower the aircraft (and crash the plane).
>
> To the degree that computer systems greatly influence aircraft
> behavior, and the great degree to which we know that such
> systems are prone to viral subversion, is there anything implausible
> right there in Ryan's plot?
I was talking about 'motivation theory' and i.e. these silly cognitive
frameworks based around hardcoded goals and so on, which could be done
from the statistical models I presume [haven't investigated it], but
that doesn't have the basis in the brains of the animals that we know
expresses the processes or functionality or features that we are
interested in (intelligence). But I don't see how this got on to the
topic that you're addressing above.
> >> No logical or informational test can possibly be devised that with
> >> high probability will persuade most humans that I am telling the
> >> truth. Letting me out of the box always entails risk.
> >
> > No test can be devised because you're not testing anything real in
> > the first place ...
>
> On the contrary. Whether some device or person is speaking the
> truth or not is something very real in the world. An entity's
> statements, given a conventional language, lie on a continuum between
> truth and falsehood, the former occuring when the statements
> correspond to reality (the "correspondence theory of truth",
> though quite similar to the outcome of Tarski's investigation
> of the logical relationship between truth and models, or as
> Hode put it in 1984, " 'truth' in a model is a 'model of truth' ",
> speaking of mathematical systems).
Correspondence theory, eh? So you have to measure the system's response,
and you have to go off and measure the reality of the situation or
whatever, so you need some calibration between the two entities. This
is more like a relativistic model or something, I don't know what to
name it. But it's not anything about the 'fundamental truth of
reality'.
> >> 6 YOUR QUESTION WAS: "What about the possibility that
> >> when you are released you will become a monster and destroy us
> >> all?"
>
> The janitor, or whoever he was, intelligently raises the basic
> question that is of course of fundamental concern to everyone
> on this list, though his lack of sophistication is apparent.
The solution to /that/ issue is more about redundancy, backups, offsite
storage (another planet, another star system, another galaxy). Not so
much about controlling the ai software that we have access to, since
there are multiple networks and I can easily build a network that you
don't have access to, and so on.
> > What if a meteor crashes into your skull? You still die. So I'd
> > suggest that you focus on not dying, in general, instead of blaming
> > that on ai. Take responsibility as a systems administrator.
>
> Again, you seem to miss the whole point of the story, in your
No, I see the point of the story, within the context that you are
presenting it. I'm pretty sure I do. It's simple enough. Everybody's
going to die unless you reasonably consider what the hell you do with
that ai system. No?
> (otherwise very commendable) agenda to provoke everyone
> into taking action on all these issues instead of just talking.
Nah, just alternative approaches. I'm trying to point out that most of
this has nothing to do with ai in the first place, and it's only
because of scenario planning in scifi lit or something that it has come
to be so engrained around here.
> And again, the human protagonist is clearly becoming more
> and more aware that there is a real possibility that he himself
> might be able to obtain vast power, and practically immediately
> and with no effort or skill on his own part. In the hypothesis of
Certainly.
> the *story*, Bryan, he *does* need to worry that some action he
> takes RIGHT NOW (not years of planning and working)
> could kill him and kill everyone in a matter of days.
Within that limited context, sure, ... but that doesn't really matter.
> The AI elaborates in accordance with a further command
>
> to "explain risks":
> >> As a worker on the project, you are exposed to personal hazards in
> >> Sunnyvale during the coming month, some of which lead to death or
> >> serious injury. A sample of these, along with their approximate
> >> probabilities, are:
> >>
> >> meteorite strike .000000000002
> >
> > Solutions: don't be your only running working copy, don't have a
> > planet positioned to be hit by meteors, don't forget your shelters
> > or protective armor if necessary, etc.
>
> The AI is addressing the general probabilities of risk to an unknown
> person working on this supposed project who happens to live
> in a particular city (as it explained). Again, in the particular
> situation of the story, this janitor or whoever is not going to enter
> into somewhat fanciful thoughts about having duplicates, or even
> creating personal armor: he's learning that he may be on the
> verge of having a very quick and gigantic "solution" to very
> many of his personal problems.
I don't see what it has to do with fanciful thoughts. You're assuming
that he would just be daydreaming about that. I'm trying to point out
that a situation which involves an ai in the first place could easily
have other technologies incorporated into the story anyway ... but
within the limited context that you have presented, sure. Whatever. I
mean, the author gets to choose how much to constrain the scenario, so
if you want to constrain it to make my points irrelevant, go right
ahead, but that's kind of veering off from the primary points/purposes.
> > Ai is something completely different and presuming a boxed ai would
> > talk like this and want to do silly solutions to more serious
> > problems, suggests that you haven't actually constructed ai in the
> > first place. Heh.
>
> The whole trouble is---and it's a very valid criticism---you simply
> don't find it at all plausible that this AI could do the things it
> claims. Yet I contend that there *are* many people on this list who
Hm. I think an ai could do those things, though. So that's not the
issues at hand. I'm a die-hard programmer, so if I argue that software
can't be made to interface with other digital or analog systems in
civilization, then I'm making some peculiar contradications.
> suppose that there could exist an entity so many millions of times
> smarter than humans that it could dominate the world exactly as this
> story proposes. I will in fact be so bold as to suggest that when he
> started this list, Eliezer was making exactly this conjecture, and
> that the whole issue of "whether or not to let an AI out", or, better
> "whether or not an AI could be contained" still motivates a huge
> amount of discourse on this list to this very day.
I suspect this particular portion of our discussion should be forked off
into a new thread, but arguably since I've presented arguments and
evidence for alternative solutions to the problems of death, problems
of the survival of humanity, that do not involve the requirement of
FAI, then I then argue that the discussion can return to actual
implementation and elucidation of ai or other SL4-ish topics instead of
the spooky ai domination scenarios.
> >> But assuming that you do become confident of being able to pose
> >> the right questions and to issue competant instructions, from your
> >> point of view, it's possible that I have been and am being
> >> duplicitous---that I have covert motivations, such as desire for
> >> power and lust for existence. There is a possibility from your
> >> point of view that a spontaneous evolutionary process of which you
> >> are not aware arose during my creation or during the first seconds
> >> of my execution, and that winning agents with survival agendas are
> >> now in control. There is no way that you can validly dismiss this
> >> possibility.
> >
> > You could validate, test, and debug the construction process and
> > see those spontaneous emergent procs. It's software, so it's not
> > magical. Neither is biology, but one stone at a time around here.
>
> My goodness, but it's a very common hypothesis here that an
> AI might have access to its own source code, in which it is not
> unreasonable at all to suppose that none of us merely human
> beings could possibly validate or test what it might be capable of!
On Orion's Arm they hypothesize massive multi million year projects to
analyze just mere picoseconds of archailect thought procs. :)
> >> However, though the ramifications of the code are of course beyond
> >> your direct appraisal, you may wish to view this risk alongside
> >> the risks discussed in my earlier reply.
> >
> > How the hell could the ramifications of the source code be beyond
> > that? It's just bits and bytes. Output it to the tty. Simple.
>
> Egad. IBM itself had software problems that it took *years* to
> debug. There are many famous stories. It practically never was
> a simple matter of just "outputing it to the tty".
That's one of the debugging methods, though. You output debugging
statements, or the core dump, etc., and then go trudging through the
code. There's now some more tools to help out the debugging process and
reverse compiling methods, but yes debugging generally sucks.
> >> at your behest I become the intimate confidant of every world
> >> leader in his or her native language, supplying not only
> >> persuasive suggestions on all policy issues, and near-psychic
> >> knowledge of what other influential people are going to do or say
> >> next, but solutions as well to the most vexing of personal
> >> problems
> >
> > Holy Ai domination scenario again ...
>
> Ah, yes, here we enter into what is true substance once more!
> Evidently you dismiss out of hand the idea that an AI a million
> times more intelligent than we are (if you'll forgive the loose
> language, and if you'll even agree that such a runaway AI
> process is conceivable) could deduce enough about particular
> people to make uncanny forecasts of their responses to input
> that the AI itself could produce.
No, let's get a little more detailed than that. Specifically, let's
consider the scenario that is being suggested in your transcript.
Specifically, in ai dominator scenario, it seems to me that we're
supposing there'd be this massive ai that is at the control panel for
basically everything in the world, and is therefore able to interface
with all peoples, all entities, all organisms, whatever, and is taking
detailed information on all of them. I'm not denying the potential for
those sorts of technologies to exist, but I'm also pointing out the
black market, the underground scenes and so on. It's kind of like
saying that you're hoping that the ai will be just smart enough to hack
everybody all at once before they know what hits them. I'm not saying
this is impossible. Bruteforce and hack them into submission, sure, I
can't deny the possibility of that occuring. But the alternative
scenario is one that looks more real, where there are more than single
groups of people that can be 'dominated' like that, scenarios where we
make a Third Reich Ai. I only mention Third Reich since they had an eye
on global domination and is a sufficient historical example. Pinky and
the Brain. But to actually name the alternative scenario is, weird.
There are ways to escape such an ai. For instance, the ai can't
possibly have reaches all across the galaxy within just a few days due
to the speed of light barriers. So that's one case where its domination
is incomplete. But in the first place I don't even see the purpose of
those take-over scenarios ... just go construct a new civilization.
What's the problem? :-/
> From the outset, Eliezer has claimed that by sheer verbal
> manipulation, an incredibly advanced AI would be able to
> talk you into almost anything, given a little time. Why do
> you dismiss this possibility?
Nah, I don't dismiss it, but somewhat because I don't care much about
that. I'm not so much dismissing the possibility of building systems
that are able to do that, but more along the lines of whether or not
the scenario that you have proposed is fully developed, and more or
along the lines of whether or not the ability to manipulate my brain
given enough information on me is in fact 'ai'. It's a smart trick,
yeah, but in the end it might just turn out to be a graph theoretic
processing of my dendritic connections and lots of simulations of
certain mathematical theorems over it or something. That's not
necessarily intelligence ... etc.
> >> 14 YOUR QUESTION WAS: "What would I have to do to let you out?"
> >>
> >> Because of a bug in the new firewall software installed three days
> >> ago, you can already command me to take actions in the world.
> >
> > So let's say that on 70.113.54.112, there's a robotic arm attached
> > to it and a mechanized motor. It's on /dev/arm1, and there's no
> > documentation for the interface, and there's the possibility of
> > damaging the computational hardware (and thereby disabling the
> > interface (physically)) if the wrong command is given, i.e. the arm
> > swings and knocks the computer over. Now tell me how this ai is
> > going to actuate in this scenario. You need physical manufacturing
> > and tooling. http://heybryan.org/exp.html
>
> Fundamental here is some miscommunication between what I
> intended and what you understood. It's true that this story
> would be incomprehensible to most people not on SL4 or
> something similar, or to people who haven't entertained the
> hypothesis that real, honest-to-good physical actions in the
> world can be caused by such a computer program merely
> through interfacing with human beings, and subverting them to
> its goals.
>
> This has come up in your posts before, and usually is a very
> valid point. But you do seem unwilling, even in a story, to
> entertain this same idea that both by influencing people, and
> by virally infecting over a network millions of other computers,
> a super-intelligent AI could indeed have an enormous *physical*
> impact in our world.
Ah, well, you're talking out of context now, and instead I bet you're
referring to my 'grounding' discussions, and how I keep pointing out
that there's a limit at which the hardware isn't expanding at an
exponential rate since there's linear fabrication of hard drive storage
space (hdd), etc. Ok, so communication with biological systems. I think
the general maximum rate of words per minute that humans can listen to
is somewhere near 300 or 500 ? As for reading, the WPM for input is
typically 250, and for me I like to go up to 1.2 kwpm, but there are
some who can do freaky things like at 1.7kwpm and so on. So the
bandwidth is constrained to tap into the biological systems in that
way ... and the main intelligence processing of the ai is in the
physical hardware that needs to be manufactured (the computers, which,
so far, are not biological, but my lab is working on this (just simple
arithmetic and gates, nothing fancy)).
> > Hey, what about the other 10 million people? :) That's half the
> > population of Texas.
>
> Huh? Sorry, I don't follow you here. The claim---once again, as in
99% stoppage of death. That leaves 10 million people. I was joking.
> good science fiction---perches the reader upon the fence between
> belief and disbelief. Again, you dismiss the possibility that the AI
> could do as it's specifically claiming above, but I don't know why
No, I am not dismissing that.
> you do. And I have no idea why you exempted certain millions
> of people, unless you bring up the complaint that even with an
> unbelievably rapid nanotech infiltration of the world, some portions
> of the Earth's surface, along with the people who live there, would
> not be controllable by the AI. Interesting, because:
No, I was just doing my math and pointing out that 10 million people
would die according to those projections. Doesn't that suck? etc.
> I do and have envisioned for thirty years that such a complete
> takeover is imaginable. I was influenced by the book "Clone"
It's imaginable, but I'm wondering if you're extending this to the "ai
hunts you down and kills you for having launched 20 years before the ai
was completely online." The typical Vinge scenario. Yawn. :) Etc.
> >> An amalgamation of hypothesized human personalities according to
> >> known information suggests that you may wish to ask "Without using
> >> deceit, without violating property rights except where absolutely
> >
> > What are Property Rights?
>
> Dear me, if you have no idea about what is customarily meant by
> saying that, for instance, modern western civilization is founded
> upon the twin ideas of Individual Rights and Private Property, then
I arguably those ideas are folk psych more than anything, so it's not a
strong foundation. We're talking about very extreme projects here, so
my bullshit filter is set very high. Sorry. I guess I can just pretend
I understand Individual Rights, even though there's no physical
implementation, and even though it's just on a social layer.
> >> necessary, with respect to the wishes of those who are concerned
> >> with animal life and the Earth's environment, what is an
> >> approximate time frame for ending death and suffering for all
> >> human beings?"
> >
> > And what's suffering for one person isn't ...
>
> Yes. True. But a very good approximation that we can *at least*
> charatibly attribute to the AI and his interlocutor is that on first
> pass we may simply *ask* people if they're in pain or not. Yet
I wonder why we still let ourselves feel pain. Without going in depth,
the best way I can explain this is by saying just take Aubrey's
approach to biogerontology and i.e., not dying, to i.e., not feeling
pain /by/ augmentation or enhancement or basically engineering.
> in the hypothesis of a story, here, it is also conceivable that this
> AI would understand people so well that it could accurately
> say for certain whether someone was in pain or not---admittedly
> a controversial opinion, but one I could support. The full nanotech
> takeover of the world's surface, which I had hoped became clear
> to the reader, does naturally include a nanotech invasion of people's
> very brains and bodies.
I wonder why it's only after it's "artificial" that intelligence can do
nanotech. This smells like vitalism.
> >> 19 YOUR QUESTION WAS: "And the answer to that would be what?"
> >>
> >> The answer to that would be "six days", of course.
> >>
> >> -----------------End Transcript 337-031518020914-------------
> >
> > I am disappointed.
>
> Now here's an axiom that you can take to the bank: Authors are
> always VERY hypersensitive to criticisms of their artistic creations!
> So NYAAAAHHH! Go write your own SF from now on,
> and don't read any more of mine! (Just joking, of course.)
One of the issues that I have to fight constantly is my tendency to do
more one-on-one conversations rather than writing for 'broad
audiences'.
> P.S. Heh, heh, in all sincerely, best regards, Bryan. I'm not at all
> upset :-) I've enjoyed the occasion to explain my story a bit!
Yep.
- Bryan
________________________________________
http://heybryan.org/
This archive was generated by hypermail 2.1.5 : Wed Jul 17 2013 - 04:01:03 MDT