So the story of "access to energy" is in fact the story of economic development itself, the story of how packs of primates structure themselves to shape the world using knowledge in the form a transmissible and inheritable culture. Focusing on energy, a loosely correlated intermediate output of this process, is missing the finer details of that structure. As you observe, energy efficiency is just another way a well structured society can improve their well-being despite relative scarcity of raw energy inputs - the welfare effects are primarily the results of structural reasons, not energy reasons.
The well known "resource curse", which I hinted at in my top level comment and which downvoted me to "-1", is a perfect illustration of this principle:
-> We are given an advanced technological frontier AND given a large, free resource that is relatively valuable within that frontier (which we can loosely equate with energy, since you can trade any resource on the world market for oil, gas, uranium etc. or the implements necessary to transform them into energy).
-> There is strong empirical precedent to suggest that, on the long run, the inhabitants of that particular political unit risk decreasing welfare, lower real wages, and curtailed individual freedoms, unless their society is well structured (say, like Norway)
This is why I abhor explanations like "energy poverty": if energy was the real reason, then how can getting more energy make you worse off? Why should we abstract away the very essence of the problem staring us in the face - the political system in the most general sense, culture, social relations etc. - and pick some loosely correlated variable that's neat and makes us sound insightful?