The Annotated GPT-2

So good, also for some good resources on how to train these networks:

* checkout mosaic ml, it has some recent improvements like Alibi that are important https://docs.mosaicml.com/en/v0.3.1/method_cards/alibi.html

* Huggingface trainer

* Pytorch lightning (still has some great training improvements like LR find and saving top k best checkpoints ect)

* Deepspeed/megatron/bloom

For inference:

try a few alternatives to OpenAI like https://text-generator.io/

or hosting your own network/huggingface pipelines (can use a feature extractor pipeline and a text generation pipeline to support getting embeddings and generating from the same one model on the GPU)

This is a great article for anyone working with language models!