Hacker News Reader

Comment by WithinReason | Hacker News Reader

The trade-off can be that you're not as free in designing the architecture of your model, and you have to train it in a special way. This is actually a huge obstacle to making neural networks more efficient. You could design HW that would run networks 10x-100x more efficiently, but the networks would have to be specially trained for the HW, and the SW investment to create the SW infrastructure to go with your HW is 10x the effort of designing the HW. Nvidia knows this in the area of GPUs, and they have 10x as many SW engineers than HW engineers. Neural network HW startups have yet to learn it.

View on HN

"The hardware lottery" describes roughly the same phenomenon: https://hardwarelottery.github.io/

This is one reason why I wish there were more interest in FPGAs.

The idea of DRAM with lots and lots of tiny FPGAs is an intriguing idea. I wish I were a chip designer to run with it and see where it got me.

IMO FPGAs sound like they’d be good with extra flexibility but in practice ASICs are just better. Higher frequency, lower power and there just isn’t a way to make up for the frequency gap. You can quantize and do fancier stuff, but then you need to fine tune or train on your actual inference HW to ensure you’re actually correct. You can emulate your quantization scheme and train on GPUs, but then your GPU training will be slow af. Building a true model specific bitstream for an FPGA that is actually utilizing all of the resources of the FPGA is not simple, and even if you do, overcoming the frequency gap is hard.

Can anyone explain further as a simple example?

Neural networks are commonly evaluated with matrix multiplication. A matrix multiply is orders of magnitude faster if all three matrices fit in your caches. So you want to tune the matrix size to the hardware.

In addition, once you did that (tuned your hardware to a matrix size) it can be inefficient to use smaller matrices. So even if you can make a network that uses smaller components the HW will not gain the corresponding speedup.

MatMult = the dot product, effectively. I can't believe that allowing smaller vectors is difficult or even non-trivial in hardware.

You’re assuming caches know anything about dot products.

That seems a non-sequiteur to me. Can you elaborate?

GPUs are a good example: SIMD means all 128 threads do the same work. If only 1 thread has work to do the other 127 threads still take up resources.

Ah, right, gotcha

Reply on news.ycombinator.com