Air has negligible thermal mass compared to everything else solid or liquid in the fridge. As long as you don't leave the door open long enough that fridge contents themselves start warming up, the energy losses are minimal.
Quick back-of-the-envelope calculation:
My fridge is around 1.0 x 0.5 x 0.5 m, giving a volume of 0.25 m3.
Outside temp is around 25 C, inside the fridge is around 5 C.
Air specific heat capacity is around 1000 J/m3/K.
If the fridge only contains air (it does not), and all air gets cycled out when I open the door (it does not), the fridge needs to pump out 5 kJ of heat after each door opening.
I don't know what's the typical efficiency, but let's say the fridge pumps out as much heat as you put electrical work into it (COP=2).
That's around 0.0014 kWh wasted per door opening.
It's nothing compared to just steady state consumption of a closed fridge.