The pure math world (and certain rarified branches of physics and CS) are quite different from the "measurement-based" sciences. If anything they're almost more similar to endeavors like musical composition (or even poetry) than any "lab-based" science. Particularly when it comes to the fetishization of "genius", "pure talent" and all that -- and especially when it comes to the character model (myth, really) of the lone genius who locks himself in an attic for N years and not only solves some major problem but totally reimagines the field, front to back.

I mean - at least in terms of the culture of pure mathematics, that's the image one can easily fall into the habit of having about one's self, if one attempts to do mathematics. Such that if you were to switch to physics, astronomy, or CS... you actually would feel like you were "leaving math".