It's more than that though. As someone with a background in applied mathematics (and plenty of experience in software), part of the problem when learning physics is that physicists have their own language of mathematics which makes a lot of quick short cuts. Even fairly quantitative people coming to physics will be confused and need to take some time to adjust.
A perfect example from the college level is that physicists will talk about the "derivative of a vector". Take a statistician, a mathematician and a physicist and ask them "what is the derivative of a vector" the first two will tell you 0, the third will just draw a dot over the vector. Something a simple as F = ṗ (there's a tiny dot there, but I'm not sure if you can see it), where p is a vector of momentum coming from stats, makes it look like F = 0. And it's not even as simple as "well they mean p(t)", because physicists will flip back and forth between thinking of p as a function and p as an actual vector depending on what's convenient.
So it's less a pure function of "the math is hard" but more so "physicists have their own conventions of doing math and the best way to learn is is to do math with physicists for a long time".