Inference needs very little compute, it’s mostly IO. hence the low power draw, training is compute-heavy and uses lots of power
The last numbers I saw for training a frontier model used as much energy as four fully fueled up Boeing Pegasus (of which there are 113)
The Erin Brokovich data center site describes the use as something like the lifetime use of 5-6 cars, which sounds like a lot when you're filling the tank, but hardly anything when you think about how many cars there are.
I think theoretically this is still very wasteful with lots of compute that’s getting powered and sitting unused but you are limited either by the firmware or by the GPU architecture from pushing power usage even lower without cratering performance. (no one anticipated the demand for relatively low-compute devices with lots of super fast memory)