That's only because people don't realize all the obviously unhealthful foods and mistakenly believe many healthy foods are bad for you because they contain dietary fat. A baked potato is practically as bad for you as cake, weight-wise. On the other hand, it would be extremely difficult for most people to get fat eating large portions of pure meat because any sort of meat has such a low glycemic index.

Any time my brother wants to lose weight, he goes to an all-bacon explosion diet[1] and the weight just melts off. Several people at my work went on the so-called paleo diet together and immediately started losing significant weight.

If you look at most popular diets, they generally have the core effect of deemphasizing high glycemic foods from one's diet, usually by shifting calories to meat and/or vegetables.

Personally, I prefer to shift the calories to meat because it's easier to build strength and muscle, when paired with an appropriate physical regimen. In my view, physical strength is an underappreciated and core component of modern fitness.

[1] Slight exaggeration.