Notes on training BERT from scratch on an 8GB consumer GPU

Not just a single GPU, a very middle-of-the-road GPU. It's not like it's even a 40X0. A naive spec comparison says a 4090 would halve the training time for the same result, although I'm not 100% sure it would quite play out like that. It's Amdahl's law versus more GPU RAM, and I don't know which would win.

> it appears that someone managed to train a BERT model[1] to 90% of the GLUE score reported in the original BERT paper on a single GPU in ~100 hours

Wondering what the GLUE score/training time curve would look like.

The achievement of training a BERT model to 90% of the GLUE score on a single GPU in ~100 hours is indeed impressive. As for the original BERT pretraining run, the paper [1] mentions that the pretraining took 4 days on 16 TPU chips for the BERT-Base model and 4 days on 64 TPU chips for the BERT-Large model.

Regarding the translation of these techniques to the pretraining phase for a GPT model, it is possible that some of the optimizations and techniques used for BERT could be applied to GPT as well. However, the specific architecture and training objectives of GPT might require different approaches or additional optimizations. With the help of MirrorThink.ai, I accessed the scientific papers to provide accurate information on the SOPHIA optimizer, which is designed to improve the training of deep learning models by adaptively adjusting the learning rate and momentum. According to the paper [2], SOPHIA has shown promising results in various deep learning tasks. It is possible that the SOPHIA optimizer could help improve the training of BERT and GPT models, but further research and experimentation would be needed to confirm its effectiveness in these specific cases.

[1] https://arxiv.org/abs/1810.04805

[2] https://arxiv.org/pdf/2305.14342.pdf

Memory size mostly depends on the batch size. With a smaller batch size you usually learn more per sample but it takes more time per sample. When I was trying to make a foundation model for clinical notes around 2015 with LSTM I found a batch size of 1 worked best on a CPU but a much larger batch size was ideal in GPU. That is, the CPU would not be faster in terms of samples/second if I increased the batch size but the GPU would get better. What I really cared about was calendar time not sample efficiency, the CPU did best when I optimized for sample efficiency, the GPU sped up enough with large batches that I didn’t mind showing the network more samples.

That would be only true if gamer have a relevant overlap.

While it's a nice Sidebonus for people gaming, I would argue a better point is the current Nvidia strategy to starve their line of 40* with memory and it's much easier to get a 8gb card than anything else.

I got a 4090 primarily for the 24gb.

If we want to learn to build something better than GPT-4, research that shows how to train older models like BERT using a fraction of the hardware that was used when that model was originally training is enormously valuable.

BERT is 500x smaller than GPT3, it doesn't hallucinate (it might make wrong predictions, but it won't make unexpectedly wrong predictions), it can run on a CPU in 500ms. I really like BERT style models (there are a couple newer models that better than BERT but are very similar)

> I am sure a lot of work (and time) went into this but what's the point? If the model is not as good and the original then the applications are limited and the training time is moot.

If the several orders of magnitude (two in this case) improvement can be used to train other models, that's quite a thing no?

It might not be as good on the same benchmark as the original BERT, but pretraining means you can expose it to entirely new corpuses of text e.g. medical or obscure programming languages. This has the potential to make it much easier to create domain-specific adaptations of BERT.

You usually save checkpoints at some interval. You can use them to continue the training.

It is, but pay attention to which GPU. It's a 3060Ti - good, but not exactly top of the line.

Hopefully tinygrad new drivers will fix this.

Also a lot of people were porting stable diffusion/Llama to AMD I think

For what it's worth, I think the lack of comments is due to the highly academic nature of the experiment displayed in TFA.

LLMs are all of the rage now, and the pre-training techniques used for BERT ended up laying the foundation for what became GPT pre-training runs. But I would assume even among HN users, there really are only a small handful of people interacting with this side of the NLP craze on a regular basis.

Compare that with the amount of users who probably have at least dabbled in using a programming language, and suddenly you see why the opinions fly everywhere.

A lot of the practical aspects of training neural nets are still more "I'm doing it this way because Radford et al did it" or "when you picture the problem in three dimensions, this generally seems like it should work" instead of "This was demonstrated to be the best way and I can show you it's the best way from first principles." because quite frankly that's the best MO we have when dealing with the statistical principles guiding billions of matrix multiplies for trillions of different token sequences.

I'm interested in the topic but have no understanding in depth. I like upvoting stories like these to revisit them after a day when there are hopefully some interesting comments and/or clarifications.

I really like the high signal to noise ratio in HN comments.

US is still sleeping btw.