Hacker News Reader

Hacker News

Comment by sailingparrot | Hacker News Reader

Parent Full thread

sailingparrot

> Is there a reason why the pretrained models are not included as well?

Because every one else, too, lacks the time and hardware required to replicate the training steps.

There is just not that many organizations that can pay for 100-200 thousands hours of training on A100 GPUs.

So the reason why the pretained model is not included, is that it does not exist (yet).

View on HN

There should be a system in place like Folding@Home or SETI@Home for situations like this.

For what it's worth, the architecture for seti@home doesn't make much sense any more in the modern era for most types of data processing. It's simply too slow and error-prone to transfer large amounts of data between user machines. Modern radio telescope data analysis happens at datacenters with a lot of GPUs in them, just like modern AI model training.

Okay, gotcha. Just out of interest, why is it error prone? Because of some kind of merge conflicts of the results?

No, I just mean you design your system expecting the data from machine 1462 to get analyzed at some point, but that machine belongs to some random person who uninstalled your software without you knowing about it, so you never get back an analysis. Or you have bugs that come from 100 different versions of your software running because you can't simply upgrade the software on all the machines of the datacenter.

I understand.

> you can't simply upgrade the software on all the machines

You can update a website. And a website can use the local gpu.

The break the training task in smaller pieces, parallelize and distribute it part is indeed beyond my skills.

Agreed. This feels like the next big step in open source machine learning. I would love to be able to dedicate CPU / GPU towards chunking out open-source GPT-3 / DALL-E 2 clones so long as the trained weights are shared with the world afterwards.

That feels more long-term productive to me than crunching coins.

In a way, it is similar to our approach at Q Blocks. Decentralized GPU computing for ML.

> There is just not that many organizations that can pay for 100-200 thousands hours of training on A100 GPUs.

How much is this in money?

IIRC a K80 goes for prices between 0.3 and 3 USD per GPU-hour

Pretty expensive all in all

between a a few hundred K and a few millions IMO

With no discount, one A100 is about 3$/hour, so we are talking about 600K$ for a full training. Probably closer to 1M$ since in practice you never train successfully to the end on the first shot.

Reply on news.ycombinator.com