Doing math without it is pretty ugly.

(Granted, there are other objects that may seem ugly which can only be constructed by reference to infinite things.)

You can do a lot of things without assuming that the natural numbers keep going, but it is plain awful to work with.