That said, I don't think people realize that there is literally no more large scale tier 3 (redundant) data center power in the US. Even if you are sitting on 1000 or 10k NVIDIA cards, you can't deploy them anywhere.
They also need to be in the same data center for speed. You can't just colo in 2-3 data centers to get what you want. If you want to train a large model across 1000 gpus? You're screwed.
Think you can just go to the cloud? Go try to sign up for coreweave. They are full and not taking any more customers. A lot of the other sites advertising nvidia gpus are just reselling coreweave under the covers.
Forget the software problems. There are far bigger issues and they are not getting better any time soon.