Oh, sure, it's difficult to say with certainty that 100 calories on the label will be exactly 100 calories for every person ever - but there's really not a gigantic amount of variance. Certainly it's a good enough guideline for use.
It's a matter of thermodynamics more than anything else. You can't get energy from nothing. Quantifying the exact amount of energy you'll get from a food is difficult to do precisely, but the guidelines we have are good enough for use for the vast majority of people. You can be pedantic about it, but that won't change the fact that it is that simple to lose weight, as proven time and again.