This is kind of it I think. It's not just physics that drives interesting math, and it's not just recently that this relationship holds. Math is, IM humble O, the ultimate domain-specific language. It's a tool we use to model things, and then often it turns out that the model is interesting in its own right. Trying to model new things (ex. new concepts of reality) yields models that are interesting in new ways, or which recontextualize older models; and and so we need to reorganize, condense, generalize, etc; and so the field develops.