Yes, but if you were planning on exercising just to burn calories, just don't eat them in the first place.
Fit people need to eat more. Muscle takes calories to maintain. You can't maintain muscles without exercise. Your body expects that you're running down prey on a periodic basis. If you don't meet those expectations not everything will work as it should.