Something i have personal experience with -- I knocked off 80 pounds a few years ago.
Diet controls mass.
Exercise controls fitness.
They're mostly[0] decoupled. This is not intuitive to reason about.
[0] there are some interesting things in difference in metabolic rates of tissue types, but it doesn't really matter much for mere mortals.