> retail GPUs are much more powerful

I agree with your basic point about building vs renting, but retail GPUs aren't simply more powerful. Retail GPUs are just as fast but have a lot less VRAM and memory bandwidth. The RTX4090 has 24 GB of VRAM with 1 TB/s of bandwidth. The A100 has 80 GB at 2 TB/s. For some tasks, the memory and bandwidth are the strongest constraints.