Batching is extremely powerful for optimizing power use. Your card is spending the vast majority of it's energy use fetching weights from it's memory, and it only gets to use each weight once. Running that same model on a bigger machine with lots of batching gets to amortize the cost of loading a weight across all the parallel requests. N=128 is literally about 50 times more energy-efficient than N=1.