Hacker News Reader

Comment by 0x12 | Hacker News Reader

> Even so, I fully expect memristors to deliver strong AI.

Why would they?

Is strong AI a function of storage capacity or speed?

An AI running at 1/100th of what a future AI may be capable of is still an AI and I can't see how a mere improvement of a couple of orders of magnitude would do what decades of Moore's law have failed to do so far.

If strong AI was just a matter of speed then we could theoretically take any of the large clusters available today and run it at an appreciable fraction of its normal speed leading at a minimum to a validation of the fact that it is indeed a strong AI that's been created.

The barrier seems to be more that we don't know how to go about building one from a software perspective than that we wouldn't have the capability to design the hardware.

So how would an advance in hardware suddenly fix that?

View on HN

Is strong AI a function of storage capacity or speed?

Yes, absolutely! I actually think the most appropriate benchmark is memory bandwidth, which hasn't been improving as fast as FLOPS or storage capacity. It's not a matter of running a strong AI at 1/100 speed on today's fastest supercomputer. It would be more like 1 billionth or trillionth speed.

The reason for our disappointingly slow progress in AI over the years is that our hardware is still nowhere near powerful enough to usefully implement the same algorithms as the brain, and we likely won't even develop the right algorithms until we have hardware closer to the requirements, so we can test and iterate.

> It would be more like 1 billionth or trillionth speed.

That's a possibility.

Now what algorithm do you propose to use?

My money's on Andrew Ng's deep learning research. Deep learning has already had huge success, both in reproducing the measured behavior of neurons in the brain, and in outperforming the state of the art on various machine learning classification tasks.

Here's an overview which references some very impressive results: http://www.youtube.com/watch?v=AY4ajbu_G3k

I'd really appreciate it if you could suggest why we couldn't implement similar algorithms as the brain, that possibly require a massive number of fetches & executions simultaneously (guessing this is where memory bandwidth plays in), but have the results show up much slower.

Shouldn't it be possible to have artificial AI mimicking human brain algorithms at 1/100th the speed, where perhaps a single thought based on learned information takes hours, instead of seconds?

You misunderstand. I'm saying we could, but the slowdown wouldn't be 1/100. It would be more like 1/1 billion. At that speed, it would take years to simulate a second of brain time. Not only would that be useless, it would be impossible to know if you'd actually implemented it right without being able to test it in a reasonable timeframe. That's why we'll only be able to develop brain-like AI once our computers are much faster.

Appreciate the reply. Seems I did misunderstand.

I find it hard to agree, that despite the nanosecond latency times and the terabytes of throughput we can wring out of single computing devices(gpu's etc), we couldn't simulate brain-like AI faster than a billionth of what it should be.

You're probably right though.

Memristors are the silicon equivalent of neurons -- a time-dependent function with state. In mammals, the connectivity of millions of neurons enable the emergence of intelligence. I don't see any reason why a silicon-based neuron network of sufficient size isn't capable of the same.

Now, that's entirely different from purposeful design of an AI such that it speaks English and knows what I like for breakfast. I don't think we'll ever know how to sit down and write one in Notepad.

Instead, memristor-based AIs will be evolved using genetic algorithms or other evolutionary approaches. Yeah, maybe that's wishful thinking too, but that's how I see it playing out.

A memristor can be simulated by a couple of formulas and a few variables. (just like neural networks can be simulated).

In spite of our ability to simulate this sort of process for decades we have not succeeded in building a strong AI, the difference in having a hardware version is one of performance (just like Neural Network chips are typically a lot faster than their software simulated counterparts). There is no real difference in capability here, just a (possibly very large) speedup.

Now, I'm not ruling out that such a speedup will cause us to be able to create things that so far were not possible but I have a hard time convincing myself that this will almost certainly be the case.

Note that even hardware speedup need insights to be successfully implemented. Seeing that hardware does improve at a regular pace, it looks like insights do pop up regularly.

Therefore, it isn't such a stretch to think that (i) insights may continue to pop up reliably for "a while", and (ii) not just in the domain of hardware speed-ups.

I think there's a very real minimum speed limit necessary to keep a highly connected system like your brain, an AI, or the Internet operational. Does anyone seriously expect 'TCP over Carrier Pigeon (RFC1149)' to work in practice at scale? Or that your own wetware would've developed properly if it plodded along forever below 13 Hz?

Given that biological examples of a minimum speed limit for cognition exist, and that the behavior of computational networks at various speeds seem to follow the same behavior (slower=worse), then it seems reasonable to assume that a similar lower limit for cognition exists for silicon-based networks.

Therefore, faster devices such as memristors might be just the thing necessary to get our machines thinking, and moreover, that we may never see intelligent behaviors in slower simulated environments.

I would say that strong AI could appear sooner than otherwise with faster hardware. This is because, the earliest algorithms for strong AI are likely to be flawed and imperfect. These imperfections can be compensated for by faster processing. Think of the earlier chess playing algorithms. Although Deep Blue was able to defeat Gary Kasparov, the algorithms themselves were relatively primitive compared to modern chess playing algorithms; although they achieved their goals, they did it with more computation than was necessary due to the algorithms being more brute-force orientated. As the algorithms got more sophisticated, the computational hardware needs decreased.

Faster hardware also allows you to get by without fully understanding how the strong AI algorithms work by using evolutionary methods. You use brute force evolution to evolve algorithms that are even more efficient than their parents. Some people might frown upon using evolution to create AI but that's what was used to create human intelligence so why not use evolution guided by humans to create machine intelligence?

You make an excellent point about how early imperfections in the algorithms could be fixed as they evolve through feedback, and that this step-process running faster would get us the desired (evolved) algorithms quicker.

But couldn't we accomplish this with distributed processing rather than faster hardware?

I think distributed processing is another form of faster hardware. It is certainly more operations per unit time. Evolutionary algorithms are interesting because they are able to exploit parallel processing better than most. Evolution amounts to a type of heuristic tree search, where once an algorithm has found a solution to a problem, the knowledge is persisted and shared with other algorithms.

Later algorithms can be run on lesser hardware because they don't have to retread the same territory as their ancestors; they only have to search the space that hasn't already been examined. As the saying goes, they search "smarter, not harder".

Is strong AI a function of storage capacity or speed?

Storage capacity.

The brain is relatively slow. Ignoring microtubules, we're just dumb chemical reactions. But we are a lot of dumb chemical reactions. A human brain is ~ 100 billion neurons with each neuron having 100 to 7000 connections to other neurons.

So, just estimating here, 100 giganeurons * 1000 connections = 100 trillion neuron connections.

To simulate that, we use matrices of floats for each connection. So 100 trillion 4 byte floats = 400 terabytes of memory (with lots of hand-wavy assumptions). The fanciest GPUs we have today top out around 3 GB of memory (but they have 1k great run-in-lockstep cores) and the largest whole non-proprietary system I've seen tops out at 1 TB of local RAM: http://www.siliconmechanics.com/quotes/212240?confirmation=1...

We can simulate parts (vision and hearing have advanced on neural networks in the past 5 years), but higher level cognition/motivation is still an unknown quantity.

Reply on news.ycombinator.com