Hacker News Reader

Comment by sroussey | Hacker News Reader

For neural networks, it makes sense to skip pure digital design.

When I learned how to design an ALU to say, add, and wait for the propagation of carry bits that’s like O(n) where n is the number of bits in the number, it made me want to just use superposition for addition, which is physical and instantaneous. Of course, that has all sorts of other problems that make it worse (so much worse).

Once you learn how to slow down the earlier bits, you end up with all the bits arriving at the same time, and you can have up to n adds pipelined and timed to the clock when output matters.

But with NN I think now would be a fun time (and likely the last decade as well) to rethink the basics right down to the basics. Everything need not be a NAND gate. :)

View on HN

Reminds me of all-optical diffractive deep learning. It's such a cool concept and really made me rethink computing.

https://science.sciencemag.org/content/361/6406/1004.full

This runs into the problem that the power efficiency of analogue circuitry is dramatically worse. A FET dissipates the least heat when it's fully on or fully off. Operating in the resistive regime will result in orders of magnitude more dissipation.

It's difficult to over-come that, particularly because it's not a comparison between 'analogue v 64-bit float', but 'analogue versus 8-bit int'. (it's tough for scale analogue circuits to operate even when 8-bit accuracy).

That just means an FET isn't the right device for analog circuitry. Maybe this memristor device will be more efficient.

It's a bit more fundamental than that. A FET goes between say, 0.01 Ohms and 1,000,000 Ohms for fully on and fully off. The times in between are where it passes the most power through it and that's because of how Ohms law and Power being current times voltage (ignoring differing phase between current and voltage in AC circuits). This all still remains true even for the memristors. You can do analog stuff with higher resistance and lower currents, but then you end up more susceptible to noise and other randomness induced by the environment: thermal noise, electric noise, magnetic noise, and RF noise (technically also electric noise, but more organized), etc. All that being said, for a NN this might actually be somewhat acceptable, but for general analog circuitry you can't get past this problem at all. That's a large reason why most analog circuitry these days does as much as possible in a digital domain before converting back to analog, even simple amplification. A class D amplifier does the whole job by chopping up a large voltage into just on and off and then filters it afterwards. Even though you have power loss in the filtering, the end result is much easier to have stable, and get the desired amplification. You still find class A and B amplifiers where the high frequency chopping noise isn't possible to filter out or would cause other problems, particularly when you look at RF circuitry since you'd end up transmitting that high frequency noise and causing other problems for yourself.

It's my understanding that, to some extent, nerves and muscles operate with hysteresis based on charge thresholds, and rely on pulse-density encoding for variability rather than proportional voltage or current modulation. Presumably, that seems to be why your muscles shake when under heavy strain.

It sorta makes sense, considering your body is generally electrically conductive everywhere. Signals propagate as waves of charge sustained by cells opening and closing little pores that selectively release ions. "Conductors" have pores, and "insulators" don't. That's also why nerve signals are relatively slow to propagate vs. straight up electricity. That's also why you can measure muscle activity via an ECG; you're measuring the transient voltages being generated by the waves of charge moving around. The voltages themselves don't have much meaning to your body. Of course, if you apply enough voltage to build up a charge inside a muscle, you can cause it to actuate.

I recently had to get a pacemaker, and apparently the particular lower threshold to trigger my heart muscle to contract is around 0.7V for 350uS, at whatever impedance the lead happens to be. Below that, and my heart muscle does nothing. Anything above that, any my heart muscle does a full beat. The device applies a 2V pulse so that there's plenty of margin, and can go up to 5V if needed, in case the lead's impedance increases. The lower the voltage the better, in terms of battery life. The cool thing is that the device can safely coexist with any natural electrical activity my faulty nerves may have, since the muscle simply responds to whichever pulse happens to arrive first.

Ideally each analog device that simulates a neuron is equivalent to tens of thousands of digital transistors that simulate a logical neuron (in space-time for GPUs or CPUs and just space for neuromorphic asics) . This makes the trade-off much more feasible. .

We don't need to go fully analogue though, going asynchronous might already be a step up. In neural networks some groups do research into so-called spiking neural networks, which you could think of as electronic circuits operating asynchronously on binary signals. They are very energy efficient, but nobody really knows how to use them yet.

Clockless CPUs have been a theoretical idea for over a decade before the "neutral network" became a common phrase. Felt like a decade ago it was still a pipe dream. Resolving the asynchronous signals becomes a chore. The world feels stateful and ordered to humans. In the same way functional programming can be difficult, getting this right is very hard.

There had been purely asynchronous CPUs before [0]. Then automated clock gate insertion made it possible to approach the gains of asynchronous, but still designing in a synchronous paradigm.

[0] https://en.m.wikipedia.org/wiki/AMULET_microprocessor

Greenarraychips are clockless: http://www.greenarraychips.com/home/products/index.html

Very interesting chips.

> the power efficiency of analogue circuitry is dramatically worse

The human brain would like to have a word with you.

At the slow "clock speed" the brain runs at, it produces a lot of heat. A quick search turns up 10-20 watts as a rough estimate. It's still petty competitive given what it's capable of.

Good thing we have liquid cooling

Why do you say slow "clock speed"? Can we even use clock speed w.r.t brain? Genuinely curious.

I assume the comment was referring to the duration of the action potential and subsequent refectory period[0], which are on the order of milliseconds.

[0] https://en.wikipedia.org/wiki/Refractory_period_%28physiolog...

That makes no sense because the brain is not digital (it's "pulse-based"). There is no signal synchronizing all neurons in the brain, which is probably where a lot of the power savings come from too.

It's a very rough analogy. The brain doesn't work like a computer chip, but it does work much slower than modern chips in neuron firings per second.

A bottom-up rewrite of modern computer engineering for performance and capability would probably qualify as one of our largest engineering endeavors in human history. But I would totally be on board for that. There would inevitably be some really interesting discoveries from it, no doubt.

Here is an attempt https://dspace.mit.edu/bitstream/handle/1721.1/78468/8340892...

If you're building adders with O(n) carry times you're doing it wrong - normal carry lookahead logic is ~O(log N)

The gate count for addition is at least O(n) though.

You should note the O(log n) delay growth is for a logic (ideal) model. Considering the 2D geometry of real circuits, there are models with more detail, in particular VLSI models. [1]

I suspect due to routing and path lengths even the delay would turn out to be O(n) in VLSI, or at least a power of n (with a small constant).

[1] Seems like a good introduction: http://cs.brown.edu/people/jsavage/book/pdfs/ModelsOfComputa...

Oh yeah definitely > O(n) - but we often trade gates (and area) for speed either by pipelining and/or more parallelism (which is what carry-lookahead logic is)

Reminds me of ternary computers. I can’t think of the exact circuit type now but there exists one in three states that can make every other.

Also: https://en.wikipedia.org/wiki/Ternary_computer

I totally agree with this. I've long dreamed of something like a FPGA for NN's where the gradient is built into the hardware configuration.

Someday I'll sit down and actually try to design a prototype for such a thing.

It's also worth mentioning that the hardware is not the only thing that should change. Cognition seems hard to automate just using instructional directives for producing the states of such a machine. Encoding the gradient right into the hardware would be cool, but the optimization procedure should also move to being something intrinsic to the machine.

> When I learned how to design an ALU to say, add, and wait for the propagation of carry bits that’s like O(n) where n is the number of bits in the number

No, it's O(1). The bit size of how you represent a single datum does not grow with size of your model.

That doesn't really contradict what he said. O(n) where n is a positive constant is O(1).

Reply on news.ycombinator.com