For reference, a single 4090 GPU has more FP8 flops than the top supercomputer in 2007. A 4x 4090 computer (something you can buy today for ~10k) would be better than to top supercomputer in 2010 [1]. There's a reason deep learning only started to really work in the past few decades. We had the "theory" for a while, but no compute to actually put it in practice. And the current models are being trained on 10s to 100s of thousand of enterprise GPUs, that make the 4090 look like a toy.
[1] - https://timdettmers.com/2023/01/30/which-gpu-for-deep-learni...