My point is that a lot of people misinterpret conservation of energy assigning it a direction of causality it doesn't have. It's a constraint between variables (not an assignation as in programming languages) which are under complex feedback control systems in the body, and that's why the kind of calories you eat (fats vs carbohydrates) DOES matter, you don't eat abstract pure energy but as you say mass with chemical energy that has to be metabolized.
That's why starvation diets (just counting theoretical calories, which are numbers obtained in laboratories, dismissing the type of macronutrient) often fails in the long term and leave serious damage. The book talks long about this.