I think of it more as, math is ultimately about symbols. Like, if a mathematician says that "2 = 2" is a true statement, a reasonable onlooker might ask "Does that mean that all twos are interchangeable? Or that there's a unique concept called two and it equals itself?" And the mathematician replies, "Neither! It means that the string of symbols '2 = 2' is reducible to the symbol 'true', given certain axiomatic symbolic transformations. Nothing more, nothing less!".
And obviously we can project concepts onto the symbols, like "integer" and "real number", and talk usefully about them, but those are the map and the symbols are the terrain, as it were. At the edge cases where we're not sure what to think, we have to discard the concepts and consult the symbols.