My point is that as a practical matter, they're right. Calculus works really well. But consider for the moment the question of why you are convinced that induction is valid. There is no proof of induction's validity. The epsilon delta thing is a definition, not a proof, and serves, effectively, to invert infinity into an infinitesimal. I am on firm ground with these statements, like it or not.
I agree there's little to be gained examining what things really mean when it comes to meaning itself - it is always going to be circular. I believe that the foundations of math lay primarily in its utility, and secondarily in the social fabric of experts (which may or may not be related to utility). The eerie, beautiful way in which math describes the world is fundamentally a human phenomena, and it's based on aesthetics, not logic. I suppose my objection is simply that one shouldn't go around thinking that the epsilon delta definition removes the ambiguity and messiness that actually underlies mathematics.