Maybe it's the differential structure itself? Or an (over?)emphasis on the study of functions from space to space instead of the study of the space itself? Or the specificity of calculus (the study of one specific space) instead of the generality of topology (thinking about a lot of spaces and comparing them to each other)