It specifically matters because the number influence on a person's weight is calorie intake/calorie usage.

Other things might influence to a degree, but the main problem is always going to be caloric in nature.