From: Eliezer S. Yudkowsky (sentience@pobox.com)
Date: Wed Feb 06 2002 - 00:04:08 MST
> Mitch Howe wrote:
>
> I'm in the process of beefing up the Sysop Scenario FAQ, but I'm stuck
> on the question "Could someone hack into a Sysop?"
>
> The only answer I have is "No, because if its design were so vulnerable
> a Friendly SI would never implement it." I believe that this is a valid
> answer, but such ontological responses are not very satisfying to the
> reader or the writer.
The response I would give is this:
Any hack requires a vulnerability. Human code often contains
vulnerabilities because humans have relatively short attention spans and
can't linger for hours over each line of code. Also, humans often write
lousy code because we are basically hunter-gatherers and programming is
not one of the tasks we are evolved to accomplish. It's quite possible
that code written by a sufficiently smart AI simply would not contain
code-level flaws, such as buffer overruns and the other flaws that modern
hackers exploit. To put it another way, the "game" of hacking doesn't
necessarily continue between superintelligent programmers and
superintelligent attackers. It could, but it's also possible that given a
superintelligent programmer there are just no mistakes left to exploit,
even if the attacker is a hundred times more superintelligent.
-- -- -- -- --
Eliezer S. Yudkowsky http://intelligence.org/
Research Fellow, Singularity Institute for Artificial Intelligence
This archive was generated by hypermail 2.1.5 : Wed Jul 17 2013 - 04:00:37 MDT