What you missed out on is feedback loops that affect your basal metabolism. So: eat less, your body decides it is starving and reduces your basal metabolism, you continue to eat less (better-than average willpower!), and... you gain weight.
I'm a physicist, and I roll my eyes whenever anyone starts up with simplistic arguments about humans, food, and thermodynamics. Thermodynamics holds, but the system doesn't work the way you think it does.