Mathematics is a part of physics. Physics is an experimental science, a part of natural science. Mathematics is the part of physics where experiments are cheap.
— V.I. Arnold: "On teaching mathematics" (1997)
— V.I. Arnold: "On teaching mathematics" (1997)
Just like English isn't Physics even if you can have describe aspects of Physics using English.
For Math and English to be Physics, it needs to specifically define what physical meaning the variables and parameters involved have.
Typically, you recognize it as Physics when it comes with units.