"I ate 500 calories of potatoes per day and lost weight. It must be because of the potatoes."

Considering even sedentary energy requirement is about 1800 calories per day, I have a shocking alternative theory.

Eating less makes you lose weight.

This concept of less-calories-in regardless of where they come from is repeatedly supported by science, read any fad diet article on Skeptoid: https://skeptoid.com/episodes/4664