In my own impression, math is best done with total disregard for applicability, which is to remain entirely accidental.
Concerning "The problem is epistemic: why is mathematics, which is developed primarily with aesthetic considerations in mind, so crucial in both the discovery and the statement of our best physical theories?"
Abandoning the primary goal of "aesthetic considerations" would undoubtedly lead to such math becoming unusable, or even degenerating into non-math.
Therefore, I believe that in everybody's best interest math should stick to its own goal of exploring mere aesthetics.