I was glad to see the oft forgotten, and correct view in my judgment, Aristotelian position that infinity is never actual, always a potential. The number line is open-ended not "going to infinity" as the flawed modern approach puts it.
The set of integers is the first infinity in ZF. Maybe you could reason a bit about integers without building a model of them, but you wouldn't have a set theory and you wouldn't have access to algebra.
I think you are begging the question by accepting ZF (and that basis for algebra)and then rejecting any possible alternatives by saying you'd lose algebra. This is not true, even the article mentioned that much of math can be founded on infinity as potential, just a different approach. The issue really is a question of philosophy and what are axioms, what is knowledge and what is math.
There's no way to know which axiom is "correct", because you can't actually build a physical infinite set.