11B is a chonky model! Curious to hear what kind of hardware you used to train it. GPT-J (a 6B model) was trained on a v3-128, I think, so this is about 2x that.
https://www.dell.com/en-us/shop/nvidia-ampere-a100-pcie-300w...
Do we have rough estimates of the training GPU compute needed to train GPT-3 and has OpenAI said anything about compute resources used to train ChatGPT?