I'm not a physicist either but this stuff isn't magic. Most of the mathematics used by physicists isn't complicated if you've managed to get past calculus.
I do agree that you can go quite far with calculus, linear algebra, and probability. But I do think that you overstate the case.
¹https://webhomes.maths.ed.ac.uk/~v1ranick/papers/eisenbudhar...