Hacker News Reader

Comment by amelius | Hacker News Reader

One thing I'm noticing lately is that these DL libraries and their supporting libraries are getting unwieldy large, and difficult to version-manage.

In my mind, DL is doing little more than performing some inner-products between tensors, so I'm curious why we should have libraries such as libcudnn, libcudart, libcublas, torch, etc. containing gigabytes of executable code. I just checked and I have 2.4GB (!!) of cuda-related libraries on my system, and this doesn't even include torch.

Also, going to a newer minor version of e.g. libcudnn might cause your torch installation to break. Why isn't this backward compatible?

View on HN

The complexity of deep learning algorithms is low but the complexity of the hardware is high. The problem solved by these gigabytes of libraries is getting peak utilization for simple algorithms on complex and varied hardware.

CuDNN is enormous because it embeds precompiled binaries of many different compute kernels, times many variations of each kernel specialized for different data sizes and/or fused with other kernels, and again times several different GPU architectures.

If you don't care about getting peak utilization of your hardware you can run state of the art neural nets with a truly tiny amount of code. The algorithms are so simple you don't even need any libraries, it's easy enough to write everything from scratch even in low level languages. It's a fun exercise. But it will be many orders of magnitude less efficient so you'll have to wait a really long time for it to run.

While I think you raise important points about the dominance of hardware optimizations, I think you're massively overstating the simplicity of the algorithms.

Sure, it's easy to code the forward pass of a fully connected neural network, but writing code to train a useful modern architecture is a very different endeavor.

I disagree. I mean it's not trivial but it is completely within reach of a single person. The only part you'd really need to lean on libraries for would be data loading (e.g. jpeg). The core neural net stuff really is not that complex, even in the latest architectures like transformers or diffusion models. Look at stuff like George Hotz's tinygrad or Andrej Karpathy's makemore.

I disagree, the burden is almost exclusively maintaining fast implementations of primitive operators for all hardware. These ML libraries are collections of pure functions with minimal interfaces. There's very little code interdependence and it's not particularly difficult to implement modern algorithms to train networks.

full stable diffusion in <800 lines: https://github.com/geohot/tinygrad/blob/4fb97b8de0e210cc3778...

autograd in <30 lines: https://github.com/geohot/tinygrad/blob/4fb97b8de0e210cc3778...

Adam in <20 lines: https://github.com/geohot/tinygrad/blob/4fb97b8de0e210cc3778...

Ok, is there any way to trim down the amount of code used without reducing the performance of my particular application, and my particular machine?

I have the feeling that it's an all-or-nothing proposition. Either you have a simple CPU-only algorithm, or you have several gigabytes of libraries you don't really need.

Also, in some applications I would be willing to give up 10% of performance if I could reclaim 90% of space.

CuDNN is only for Nvidia GPUs, and those machines generally have decent sized disks and decent network connections so no nobody cares about a few GBs of libraries. There are alternatives to using CuDNN with much smaller binary size. Maybe they can match or beat it or maybe not, depending on your model and hardware. But you'll have to do your own work to switch to them, since most people are happy enough with CuDNN for now.

The real problem with deep learning on Nvidia is the Linux driver situation. Ugh. Hopefully one day they will come to their senses.

It's not just disk size. Also memory size, and loading speed.

Yes, I agree about the driver situation.

The disk size of the shared library is not indicative of RAM usage or loading speed. Shared libraries are memory mapped with demand paging. Only the actually used parts of the library will be loaded into RAM.

Reply on news.ycombinator.com