There's about 5 significant things that go into perceived temperature and comfort: Ambient air temperature, radiant heat, conductive heat, humidity, and air circulation.

So for instance, in an un-insulated shed, you could bring up the ambient air temperature with a space heater, but still feel cold because the walls are cold and not radiating heat back at you. (It would be convenient, but incorrect to say that they are radiating cold at you).

If you insulated the walls and ceiling, but not the floor, you could have a warm space, but cold feet due to conductive heating (of the floor, from your feet).

Humidity and air movement is more important in cooling, because the human body cools itself with sweat; but humidity in a structure has to be controlled, even if you want more humidity in the wintertime, it will condense and cause rot.

So basically I'm saying you want the insulation just for comfort even if it didn't make economic sense.