If that were true, math would be useless, and nothing more than an esoteric artform.
The true power of math comes from the correspondence between those symbolic transformations and observation from the real world. Two objects that look alike can be placed in juxtaposition with any other (different) two objects that look alike, and no matter how much we move them around, as long as we don't add or remove any objects, they can still be placed in the same juxtaposition as before (while this description may seem verbose and clumsy, in the real world it does not need a description - it is a much more primitive sensory perception, learned at an early age).
> obviously we can project concepts onto the symbols, like "integer" and "real number", and talk usefully about them, but those are the map and the symbols are the terrain
It wouldn't be "obvious" that we can project concepts onto symbols, if we didn't discover that symbols correspond to concepts and that symbolic transformations can help us predict the future. Thus I'd say it's the other way around: symbols are the map that we know how to read - of the terrain that we can't traverse easily.