At this point, numerous exotic mathematical tools from Langlands program, Lie algebras, algebraic geometry and topology have been used in physics.
I do agree that you can go quite far with calculus, linear algebra, and probability. But I do think that you overstate the case.