I believe our difference is that you are presupposing that aliens will have essentially the same experience with reality that we do. I agree that if somehow a separate human civilization evolved in another galaxy, their math would likely look similar to ours. But you can imagine aliens that have very different experiences of "what actually works in the real world" (due to experiencing a different set of physics, such as at a different physical scale) and thus come up with different math.