I'm honestly baffled at your surprise... say, if you crop an image, and 2 seconds later you enlarge it to its original size; do you expect to get the inital image back? Or a uniform color padding around your crop?
Scaling is just cropping in the frequency domain. Behaviour should be the same.