Of course for DL, people routinely use reduced precision as you have alluded to. There, we're looking at exascale at the moment. The second machine mentioned in the article (Summit) has ~3EF of Float16 performance (of which their application reached ~1EF). For comparison (and assuming about 50% MXU utilization - which is about the maximum I've seen), you'd need about 200 TPUv2 pods (at $384/hour/pod for the general public) to reach the same scale on this problem. Now Google does have that scale, but it's certainly non-trival.
Half-precision petascale is fairly routine these days. You can get that on the cloud for < $100/hr and I do routinely spin up such systems for ML training. However, petascale fp32/fp64 and exascale (b)f16 as discussed in the article are still fairly rare and usually preceded by months of planning to make sure things go right and the computer power is used usefully.
Disclaimer: not involved in the work, but working closely with the folks who are and I've used both the mentioned systems myself.