There is a way to look at mathematics as just a bunch of rewrite rules for things on paper. It might not be particularly inspiring, but it's a valid way to look at things.
https://en.wikipedia.org/wiki/Herbrand_interpretation
The point of mathematical semantics is that for any given theory, we can have other interpretations that don't just interpret symbols as themselves.
So we could conceivably imagine an interpretation where ∞ doesn't just mean literally ∞ and nothing more.