Why has the simplistic traditional advice worked for some? Is there big variation in the extent to which different humans are affected by macronutrient ratio and food quality? Are nuanced dietary dos and don'ts a domain reserved for the unlucky?
As for the starvation study in question, it doesn't even support his argument. the subjects lost considerable weight and their weight loss slowed because they had less body mass (and thus a lower caloric maintenance)
it really is just about being in a moderate caloric deficit and exercising.... but that wouldn't be a billion dollar industry.
So what causes people to stick with the plan?
Even if you naively assume a human to be a closed system, this is a very inaccurate claim.
You realize this statement makes no sense right? It is also incongruent with your latter statement. Calories are merely units of measurement of energy. If you eat too much (compared to what your body uses), those energy will be stored as fat.
Yes, I am aware that how fast your body sheds fat is genetic, as well as how hard your body holds on to fat cells. That's all kinda part of your TDEE. If your body is the kind that holds on to fat cells more, tough. You just have to eat a lot less, and move a lot more.
Less and more is defined by how much your body uses energy. If your body uses only 2000 kCal a day, then if you want to lose weight, you simply eat less. If you have a body that dictates you use 1600 kCal a day, then less is simply less than 1600.
It's really that simple (well, in writing... in practice, the discipline issue definitely comes into play)
I have thyroid disease, and I gain weight easily, I am used to eating little to no food (in the literal sense, because of other problem I have, unrelated to the thyroid disease, I can forget to eat and sometimes don't eat for an entire day or two), people always ask if I am on a diet even when I am just eating what I eat normally, also I love restaurants that charge the food per weight, because it saves me lots of money.
Still I am 130kg right now, and struggling to get any lighter, my body just figured how to be very efficient, for example my temperature is lower than normal (the lowest I measured in a "healthy" day was 35 degrees), the muscles related to walking are stronger than everything else (when I go to the gym, exercises that use those muscles bore me, I can keep using more and more weight until it is dangerous to the integrity of tendons and bones), while all my other muscles are very weak (my little sister can carry weight much more easily than me for example).
And also my body don't "waste" energy maintaining certain parts of it, for example hair (my hair is constantly breaking), nails (they grow slow and shatter easily when I try to trim them), muscles (the only kind of exercise that makes me stronger is very low rep high weight exercises, cardio exercises make me lose lean weight very fast instead).
Also one of the first persons to notice I had a thyroid problem was my dentist, I had a jaw problem that had to be fixed using braces, she noticed the progress speed on my case was odd, my healing speed was weird.
My parents dog accidentally scratched me with her claws two months ago, it was a very superficial thing, I only noticed later (it was not painful when it happened), it still don't healed properly.
Although I am a sort of extreme case (sort of, because there are people with cases that are much worse than mine), I can tell you it is very possible to keep a 100kg body with 2000 cal diet.
Everything else you said is observational and irrelevant when comparing yourself to another individual as there are many many things that could explain those differences.
When I was younger I could just start running 3-4 days a week and lose 20lbs over several months. Now that I'm older, and more out of shape, I've resorted to a lot of walking, plus I'm adding a spinning bike. The Apple Watch's ability to monitor my activity got me started getting me moving enough to see results.
https://h4labs.wordpress.com/2015/08/27/apple-watch-8-weeks-...
I'm hoping to lose another 20lbs this year.
And after a while it becomes a lot easier, because you'll get pretty good at estimating how much calories something has. (That omelette I got at that french restaurant? Probably 700 calories, since it was really cheesy... etc.) Plus, you're just as likely to overestimate calories as underestimate, so over the course of a given week any inaccuracies probably even out.
I lost 50lbs by counting calories years ago. It took me a year to drop all of them, but I've mostly kept it off for 2 years more. ("Mostly" because I stopped counting several months ago and gained about 10 lbs since my lowest level, so I'm counting calories again and am back down to +5 or so from my lowest and dropping.)
[1] "Lose It!" is my favorite, but I've heard good things about MyFitnessPal, which has a crowdsourced calorie database. For any food with a barcode you can just scan it, and there's a huge database of restaurant foods, as well as estimated calorie amounts for anything you're likely to cook.
What was eye opening for me was just how little 2000 calories is. We've gotten so used to large portions that an actual day of 2000 calories takes a while to get used to.
Examples: take a single donut, 400 calories. And that's for a "normal" size not the giant 8 inch cinnamon role donuts you see at most places in the USA selling donuts. Same with muffins / cupcakes. Yes I know that's junk food but I see so many coworkers eating them every morning and the odds of them staying under 2000k for that day are close to zero.
A sandwich on square bread, the old standard size, is 500-600 minimum but most sandwiches in the USA are on rectangular bread 1.5x the size of square slices which means the sandwich is going to be at least 1.5x the calories.
I'd even go so far as to say it's nearly impossible to eat 3 meals and not go over 2k. Obviously it's not impossible, but once you know the calorie counts each meal is going to be pretty small by typical USA standards.
Also, AFAIK the number of fat cells you have is pretty much constant through adulthood. When you lose or gain weight is just those fat cells becoming smaller or bigger.
http://www.ncbi.nlm.nih.gov/pubmed/25074273
https://www.23andme.com/you/explorer/snp/?snp_name=rs1421085
has a wider collection of the literature and links to the recently published http://www.ncbi.nlm.nih.gov/pubmed/26287746 which finally establishes this as the causative snp and explains the method of action.
http://onlinelibrary.wiley.com/doi/10.1038/oby.2008.318/abst...
Which is a much simpler explanation.
And even so, carriers of even the highest risk form of the gene are still on average only 6.6lbs heavier than normal: