Comment by TheLoneWolfling | Hacker News Reader

Comment by TheLoneWolfling | Hacker News Reader

Parent Full thread

TheLoneWolfling

I have no idea where this idea - that AIs would be trivial to duplicate - started, but no: I do not believe that an AI would be "trivial to duplicate". I personally don't think that an AI could be transferred between hardware at all, or at least not without some major effort, much less copied non-destructively.

There was an experiment a while back where someone applied genetic algorithms to design things on an FPGA board. They got results that tended to work, but did things... weirdly. Things like picking up clock signals from the PC used to control the FPGA. Using things that are quirks of the current enviroment, but that don't work as soon as you change anything.

I believe that an AI would probably be along the same lines. I believe that a large chunk of an AI's "core personality" would be how code interacts with the bits of hardware that aren't deterministic. Which processor core updates memory first, how exactly processor core temperature changes when the processor workload changes, the characteristics of the microphone, etc, etc.

For that matter, how would one even copy an AI? I would assume that a "true" AI would be using all the resources at its disposal. How would you copy the current registers if all RAM is occupied? How would you do a copy of something that is constantly changing? How would you copy the data currently in the processor cache? Branch predictor? Interrupts?

I suppose that part of this is that I have always assumed that an AI would be at least partly self-modifying code.

That was a really fascinating experiment. Here's a pretty approachable writeup, if anyone is interested:

http://www.damninteresting.com/on-the-origin-of-circuits/

Why do you assume AI will rely on hardware so extensively and not work in software?

> How would you do a copy of something that is constantly changing?

You pull the plug on the hardware.

All software relies on hardware. (parent (parent)) is imagining AI software as being grown, not written, and sees a strong likelihood that such "grown" software will rely strongly on emergent properties of its hardware substrate. Circuits "grown", actually evolved by means of a genetic algorithm, on FPGAs, are known to behave in this fashion; see the "On the Origin of Circuits" cite for examples at small scale -- in short, you can, with an FPGA as substrate, evolve a circuit capable of distinguishing 1KHz from 10KHz tones, but what you end up with is so tightly bound to that particular environment that, when you "pull the plug" and copy the FPGA configuration from the chip where it evolved onto another chip from the same production run, it doesn't work at all.

Of course, it's reasonable to question whether the technique described in the cite could possibly scale to "growing" something like a general AI, or at least whether it could do so in any remotely practicable span of time. But among AI theorists it's a fairly common surmise, and given results already seen for the technique, it's entirely plausible that a general AI grown by such methods would be so intimately bound to the hardware where it "grew up" that, even assuming you could copy it from its "home machine" to another machine of identical configuration, there's no reason to assume the copy would work correctly, or indeed at all.

(And all of this goes to the question of where precisely you draw the line between hardware and software, anyway. Into which category does a given FPGA's configuration of gates fall? Does your answer change depending on whether that configuration has origin in a Verilog file, or in the result of a process of directed evolution? What about the configuration of neurons in your brain on which runs the consciousness that calls itself "I"? I don't pretend to have definitive answers, or indeed any answers, to such questions as these, but they are certainly worth considering in such a discussion as this one.)

> Why do you assume AI will rely on hardware so extensively and not work in software?

Three reasons:

First, hardware is much faster than software. By the time you've built a virtual machine of some sort you're talking about (an) order(s) of magnitude performance loss. We're still speed-limited in how accurately we can emulate the SNES, for example. Sure, there are cases where it is not too bad of a performance loss, but in the case of something like an AI, I would have to assume that a large chunk of the "core" AI would be self-modifying in some form, and self-modifying code just kills most interpreters/JITters.

Second, any sufficiently complex software will have bugs. And in the case of something like an AI, I would have to assume that a large chunk of the "core" AI would be self-modifying in some form, and as such will probably hit upon some undocumented feature/"feature"/bug of the sandbox and start relying on something instance-specific. Machine learning algorithms make great fuzzers. (I know - I've actually had this happen myself. Was working on a fairly simple robot arena machine-learning "demo", and the winning robot exploited a bug in the arena to mess up other robots. (Fairly basic stack-based VM, robots could run X virtual instruction subticks in a game tick. Some instructions took multiple subticks, and there was a bug where starting an instruction [number of subticks in the instruction]-1 subticks from the end of your turn would execute the instruction on your opponent's turn, pushing it onto their stack.)

And third, I don't see how one could build something that doesn't at some point rely on hardware. Virtual machines still leak data via timing attacks. For that matter, what will you do about I/O? How do you know that the AI isn't relying on the noise on the wall in the background of the webcam for some internal subroutine?

> You pull the plug on the hardware.

Which immediately wipes all data in RAM and data encoded in the current processor state / current GPU state / etc. Great idea!

Reply on news.ycombinator.com