Over the last 3 years we made a lot of progress in efficient training methods. On text models, you can now train GPT-2 levels of coherent English from scratch for about $50 (e.g. using Karpathy's nanochat [1])
I'm sure Moore's Law for GPUs helped a little, but I'm really curious about the savings on the methodology side. (The repo mentions "hyperparameters")