Hacker News Reader

Comment by terafo | Hacker News Reader

AI training has very high minimum requirements to get in the door. If your GPU has 12GB of VRAM and your model and gradients require 13GB, you can't train the model. CPUs don't have this limitation but they are ridiculously inefficient for any training task. There are techniques like ZeRO to give pagefile-like state partitioning to GPU training, but that requires additional engineering.

You can't if you have one 12gb gpu. You can if you have couple of dozens. And then petals-style training is possible. It is all very very new and there are many unsolved hurdles, but I think it can be done.

View on HN

One thing I don't understand: If it's possible to chunk and parallelize it, is it not relatively straightforward to do these chunks sequentially on a single GPU with a roughly linear increase in runtime? Or are the parallelized computations actually interdependent and involving message-passing, making this unfeasible?

Data moving back and forth from CPU to to bus to GPU and back again for however many chunked model parts you have would increase training time far beyond what you would be willing to invest, not to mention how inefficient and power intensive it is, far more power needed than doing just CPU only or GPU only training. Back to the time part - it's not linear at all. IMO its easily quadratic.

It's not unfeasible, in fact that's how things were done before lots of improvements to the various libraries in essence, many corps still have poorly built pipelines that spend a lot of time in CPU land and not enough in GPU land.

Just an FYI as well - intermediate outputs of models are used in quite a bit of ML, you may see them in some form being used for hyperparameter optimization and searching.

Maybe a good candidate for the SETI@home treatment?

It is a good candidate. Tech is good 6-18 months away, though.

How much faster can we develop the tech if we leverage GPT-4 to do it?

Sure, but when one 12gb GPU costs ~$800 new (e.g. for the 3080 LHR), "a couple of dozens" of them is a big barrier to entry to the hobbyist, student, or freelancer. And cloud computing offers an alternative route, but, as stated, distribution introduces a new engineering task, and the month-to-month bills for the compute nodes you are using can still add up surprisingly quickly.

We are talking groups, not individuals. I think it is quite possible for couple of hundreds of people to cooperate and train something at least as big as LLaMa 7B in a week or two.

Reply on news.ycombinator.com