Hardware is not cheap either at this scale. Reallly rough math follows, please anyone correct me if I err. A100 cards are >$10,000 each. A node is typically 8 and a cluster might be 64 A100s. So about a million in hardware for the medium cluster of 64 A100s. (Cross checking, a machine from lambda labs is apparently ~175k for 8 gpus, so 1.4m for the cluster above)
Those A100 use 300watts each, plus let’s say 500watts for the rest. (8gpus300+500)8servers ≈ 25Kw * 8760 hours = 219k KwH / year. So if your costs are $0.15/KwH that’s only on the order of $32k/year.