Hacker News Reader

Comment by victor9000 | Hacker News Reader

Is there a reason why the pretrained models are not included as well? Or perhaps this is coming at some point in the future? There are many users who simply want to play around with the functionality, who lack the time and hardware required to replicate the training steps.

View on HN

As someone who has trained transformers with only ~5 layers on 1000 data point inputs for a decoder of length like 10 (so far fewer than the 12 billion parameters of DALL-E [1]), I learned two things:

1. Training is very finicky and time-consuming.

2. PyTorch model files are pretty large.

For point 1, when I was training a different autoencoder for example, I would run it for a week on 4 GPUs, only to return and see that its results are subpar. For point 2, I would get 100 MB model files for a miniscule transformer (relative to DALL-E numbers). Combined with the strong dependence of the transformer on the training data, this can make it problematic when it comes to sharing pre-trained models for tasks that the user doesn't specify a priori.

[1] - https://venturebeat.com/2021/01/05/openai-debuts-dall-e-for-...

I will gladly download a terabyte of model data if that’s what it would take, your numbers don’t scare me!

Large models generally require beefier GPUs and more RAM. Wide (not just deep) models require more GPU cores, total model size requires more RAM. Some models require multiple GPUs to train and/or run. Some require non-consumer server GPUs. These are non-trivial hardware investments if purchased, and non-trivial cloud bills if rented, even at preemptible spot pricing.

I'd pay money for pretrained models that can run on consumer gaming GPUs (8gb vram) for things like this. I wonder if anyone is selling these.

The main issue is that what neural nets are doing is just very fancy heuristic compression of vast amounts of data into a few gigs of vram (and a useful way to access it), so the smaller the model is the less it'll be able to do.

Most of the pretrained models I see for the stuff I'm working on (Coqui, ESRGAN, etc.) are really tiny in size. I'm sure the much better models are bigger but still fit under 8gb vram. It seems like there could be a market for models that fit into 8gb vram, but I'm not familiar enough with users of ML to say for sure.

> Is there a reason why the pretrained models are not included as well?

Because every one else, too, lacks the time and hardware required to replicate the training steps.

There is just not that many organizations that can pay for 100-200 thousands hours of training on A100 GPUs.

So the reason why the pretained model is not included, is that it does not exist (yet).

> There is just not that many organizations that can pay for 100-200 thousands hours of training on A100 GPUs.

How much is this in money?

IIRC a K80 goes for prices between 0.3 and 3 USD per GPU-hour

Pretty expensive all in all

between a a few hundred K and a few millions IMO

With no discount, one A100 is about 3$/hour, so we are talking about 600K$ for a full training. Probably closer to 1M$ since in practice you never train successfully to the end on the first shot.

There should be a system in place like Folding@Home or SETI@Home for situations like this.

For what it's worth, the architecture for seti@home doesn't make much sense any more in the modern era for most types of data processing. It's simply too slow and error-prone to transfer large amounts of data between user machines. Modern radio telescope data analysis happens at datacenters with a lot of GPUs in them, just like modern AI model training.

Okay, gotcha. Just out of interest, why is it error prone? Because of some kind of merge conflicts of the results?

No, I just mean you design your system expecting the data from machine 1462 to get analyzed at some point, but that machine belongs to some random person who uninstalled your software without you knowing about it, so you never get back an analysis. Or you have bugs that come from 100 different versions of your software running because you can't simply upgrade the software on all the machines of the datacenter.

I understand.

> you can't simply upgrade the software on all the machines

You can update a website. And a website can use the local gpu.

The break the training task in smaller pieces, parallelize and distribute it part is indeed beyond my skills.

Agreed. This feels like the next big step in open source machine learning. I would love to be able to dedicate CPU / GPU towards chunking out open-source GPT-3 / DALL-E 2 clones so long as the trained weights are shared with the world afterwards.

That feels more long-term productive to me than crunching coins.

In a way, it is similar to our approach at Q Blocks. Decentralized GPU computing for ML.

It literally says:

> To train CLIP, you can either use x-clip package, or join the LAION discord, where a lot of replication efforts are already underway.

Reply on news.ycombinator.com