Once you get to the point where the subjects start dividing, you'll see that the real skill that math teaches you is to look at a problem, and create a set of definitions which frame all the properties of the problem and make them make sense in that framework.
You get this skill by both playing with problems and trying to define frameworks (theorems, lemma's, axioms, ect) and by reading about the techniques that others have developed. It isn't about rote memorization, its about playing with abstract concepts till you have a way of understanding them. One thing that playing will teach you is that it will show you that the path you take to understanding something is not the same as the simplest path for explaining something once you understand it.
In contrast, in things like physics you are generally given the frameworks (by the mathematicians) and the cleverness comes from understanding how particular situations can be described by the existing (pretty abstract) frameworks (like newtons laws).
If you are bothered by how some of the ideas seem to appear from nowhere, notice that there is a lot of math out there that you don't know, and even basic statements about open and closed sets build on a lot of more basic set theory and group theory.