Replicating GPT-2 at Home
bkkaggle.github.io
bkkaggle.github.io
It's both substantially easier and faster than it was when OpenAI released their paper in 2019, thanks to both Huggingface Transformers and Tokenizers making the architectures more efficient and other companies streamlining the training process and make it more efficient for all parts in the pipeline.
You don't need a TPU cluster to train a working GPT-2 model, although it helps (unfortunately TPU support on PyTorch-based training like aitextgen is more fussy). A free GPU on Colab gets you most of the way, especially since you can get now a T4 or a V100 which lets you use FP16.
As an aside, if you do get a V100, Colab Pro is by-far the cheapest way to train an AI model. ($10/mo is much, much cheaper than $2.48+/hr on GCP normally!) Although you need to sync checkpoints to off-loaded storage in case the Notebook dies.
But others should be aware that you get what you pay for. Google still rate limited me when I used Colab Pro, and I ran into a myriad of other small problems. If that's all one is willing to spend to play with AI, 100% go for it. It's a great place to start. But if you're at all serious and can afford it, I think a local machine with a modest GPU is worth every penny.
Asking because it seems V100 performance (or the other colab paid GPU) is worth the occasional instability if you’ve set up checkpoints.
These raw numbers don't tell the whole story, of course. But IMHO, the convenience of a local 2080Ti outweighs the speed benefits of an _somewhat flaky_ V100 via Colab for day-to-day use (unless memory size is an issue, which you can't really get around).
OTOH, for just trying out stuff / one-offs, Colab is perfect - and bonus points if you score a V100.
Colab Pro isn't available outside the US (without breaking Google's terms).
e.g. in the style of a particular rapper?
If you just fine-tune on a corpus of their lyrics, you might miss the underlying poetic constraints.
If there were an additional prior (a "poetry / assonance / rhyme" model), what is the easiest way to constrain generation to respect this prior?
Thanks!
I do think it's possible to train a GPT-2-esque network to do something similar, albeit with some text encoding shenanigans.
It uses Markov chains instead of GPT-2, but the approach should work with prompt-based things like GPT-2 also: for lines that are "free" (e.g. no specific word you need to rhyme with), you can generate the line normally -- but for lines you need to rhyme with a specific word, you can just generate last-word-first and generate backwards. For a strictly LTR prompt like GPT-2, you could probably just reverse your corpus word order, generate "reverse" lines with GPT-2 given the previous line + word you need to rhyme with as the prompt, and then reverse it back to "normal" in postprocessing.
[1] https://festivalpeak.com/stylistic-rhyme-bound-poetry-genera...
Some examples of the output of this approach:
[2] https://medium.com/words-of-mimicry/kanye-west-ballade-1-a6f...
[3] https://medium.com/words-of-mimicry/me-you-and-slow-sip-slow...
I'd expect the output to be better with something like GPT-2/3, since Markov chains are so twentieth-century, but I was pretty happy at the output quality even though it often rhymed the same word repeatedly; you could improve it by weighting previously-used words, removing them from the pool of rhyming words, and/or backtracking to previous lines when you find yourself without other words to rhyme.
I have a hard time believing you can really train it with 1 V-100, unless you are talking about an extremely scale down version of GPT-2 (large).
If you can train it at all it would be with a batch size so small (probably 1?) that it would hurt the performance and it would take months.
I am out of the loop somehow?
Edit: I was thinking about reproducing the training that OpenAI did in their paper, so redoing all the pre-training, but I realized you might have been talking about training on a smaller custom dataset.
But a week of TPUv3-128 is anywhere between $10k and $20k in TPU costs alone; saying that this is an "at home" kind of experiment is cheeky at best, clickbait at worst.
The time that went into this project was almost certainly worth more than $10k.
It's not comparable to a hobby. It's comparable to paying $10k to make a sandwich.
If your hobby is building AI/ML models, a one-shot trained model isn’t going to really help you on an ongoing basis. It’s an amazing single shot project, but if your hobby is actually ML then you probably aren’t going to be happy just looking at your completed trained model - you are going to want to train a bigger, better model.
And if your hobby is building software, you can just download a pre-trained model for free.
I don’t think the analogy holds the other way.
Yeah I totally get your point about the title—the TPU quota that I got was close to about the equivalent of $20k—but in my defense I don't have any other access to compute beyond anything that I get through the TFRC or through google colab
Speaking as a hobbyist, earlier if you had enough determination you could create just about any software if you kept hacking at it long enough. CPU or cost was generally not an issue, your time and tenacity was.
This has now unfortunately changed and innovation in software (esp ML) is now largely more about how deep are you pockets are.
I can't help but feel pride and hope for the future, both the author's and the world.
Some people roll wicked real life DnD character sheets, that's for sure.
For the language model training itself I've been experimenting with a few different things. I started off with Huggingface because it's very easy to get up and running, and I still use its tokenizers library to do BPE training on the C source dataset (though there are still some hitches there – other libraries expect slightly different formats for the tokenizer model, like using different ways to represent the <|endoftext|> marker).
After prototyping the C language model training at home, I tried moving the training up to NYU's HPC cluster, which has a bunch of 4xV100 and 4xRTX8000 nodes (mainly because the sound of two powerful GPU fans running at 100% gets a bit old after a while). Unfortunately I discovered that with larger models the GPU-GPU communication overhead can be prohibitive (most of the cluster nodes only support P2P GPU communication over PCIe, which is a lot slower than NVLink), and Huggingface's implementation actually performed worse on multiple GPUs than on two 3090s with NVLink (I opened an issue track it here https://github.com/huggingface/transformers/issues/9371 ).
Currently I'm working on getting DeepSpeed running so that I can hopefully get better scaling even in the absence of a fast GPU-GPU interconnect. This is again a little bit annoying, because it seems like every framework wants a slightly different way of representing the tokenizer and training data – I've had to preprocess the dataset in about 4 different ways (plain text, loose JSON, npy (for DeepSpeed), and a custom indexed binary format for Megatron-LM). I'm also hoping to try out Huggingface's recently-released DeepSpeed integration, which (if it works) would be a really nice combination of usability and performance: https://huggingface.co/blog/zero-deepspeed-fairscale
As for other software stack hitches: so, so many. The main one is just managing the different versions of CUDA. The 3090 is only supported starting with CUDA 11.1, but many packages and frameworks only support 11.0 at best. And some of the newer things like DeepSpeed use PyTorch extensions, which require you to have the exact version of CUDA around that was used to build PyTorch. So I've had to do a fair bit of compiling packages from source rather than relying on prebuilt packages.
The path of least resistance here is probably to use the NVIDIA NGC containers, but it took NVIDIA more than a month to get them updated after the 3090 was released, and I find working inside containers for everything inconvenient anyway (I hate losing my bash history, and I always accidentally end up losing data or local changes when I exit a container).
Anyway, this ended up being a bit more rambling than I intended, but it was helpful to write it all down and maybe it'll help someone else avoid some stumbling blocks :)
I'm using 2080ti for my "at home" projects and going above CUDA 11.0 is indeed breaks lots of things - good luck trying to make something like colab/kaggle docker image for DS where you have TF+torch+sklearn+more combo.
Would you mind sharing resources you used to assemble your software/driver stack for 3090x2 setup?
https://docs.nvidia.com/cuda/cuda-installation-guide-linux/i...
This is nice because staying updated can be done through the usual apt commands, and you can have multiple CUDA versions installed by installing cuda-11-0, cuda-11-1, etc. Switching between them seems to be as easy as just making sure the appropriate /usr/local/cuda-<version>/bin dir is first in your PATH.
In general the NVIDIA documentation is pretty good, and I try to follow their Ubuntu-specific instructions whenever possible.
Pretty new to containers, so I'm glad you alerted me to those issues. Not sure which way I will go in the end but I'm on Ubuntu and trying to get Stylegan2 up… it was working great with a 1080ti.
1. Split the data up
2. Do a forward and backward pass on each GPU individually
3. Compute the average of the gradients and update the model on each GPU
4. Repeat
For step 3 you need to send the gradients from each GPU somewhere, and then send back either the averaged gradient or the updated model weights. So when the model is large (say, 3GB for GPT 774M!) that's a lot of GPU-GPU communication!
You're right that for the vast majority of ML cases, the models are small enough that the synchronization cost is negligible, though.
I wrote up some benchmarks here:
"No, we have GPT-2 at home."
GPT-2 at home: [Outputs this comment]
> Unfortunately, ALGPT-2 doesn’t perform as well as GPT-2 (ALGPT-2 gets 313131 ppl on OpenWebText compared to 212121 ppl for my pretrained GPT-2 model), but I’m writing this series of blog posts to go through everything I’ve learned over the last few months.