I don't know what a normal low fat diet is, but the overruling factor in weight loss/gain is the ratio of work (exercise) to energy input (calories).
I want to see the same person:
'immobile
'sitting most of the time, with short commutes and shopping trips
'intensely exercising 1 hr a day
'actively working out throughout the day
The point isn't that exercise _can't_ work, it's that aiming for dietary changes has a higher average ROI than exercise.
No one has argued that there is a lower than average ROI (you get paid for not eating?) from exercise.
But it's more complicated than that. Genetics cause one's predisposition as to how responsive a person is to cardio; see the University of Bath study and/or Michael Mosley's "The Truth About Exercise".
I don't think you are off, but the key is not weight loss/gain...the key is healthy weight loss/gain and that has everything to do with what you consume.
Caloric source matters, your body does different things with different sources.
Your metabolism will be reduced by reducing calories in. They are not independent variables.