All arithmetic is math, but not all math is arithmetic.
Except that if you dig deep enough, all math is arithmetic under the hood. Algebra is doing the arithmetic on variables before they're numbers. Calculus is just useful shorthand for compound arithmetic; the derivative is a divide and the integral is a sum. Exponentiation is repeated multiplication, which is repeated addition. Linear algebra is arithmetic on blocks of numbers.
So we abstract over the individual arithmetic operations for our own convenience, there are too many of them. But it's all still fancy arithmetic.
Maybe we should call math "fancy arithmetic" to help all the people who are intimidated by math...