But to be charitable, I think it comes from a place of pushing with tremendous force a pendulum of thought back from the axiomatic nature towards more exploratory and relevant mathematics. It reminds me of Howard Zinn's intro to A People's History of the US in which he says that he purposefully did not water down his arguments to be more neutral because of the overwhelming nature of the opposition.
The infinity issue is quite interesting to ponder. Obviously with our current understanding of the universe and humans, the number of numbers ever encountered by all humans over all history would seem to be finite regardless of the fundamental nature of the universe itself. There is a very strong notion of our world being discrete in that way.
And it is very interesting to pursue these ideas. For example, the proof of infinitely many integers is to assume there is a largest and we add 1. But what if the largest is unknown as is the case much of the time? This is true, for example, with computers (it can be known for the native format, but that can obviously be extended in programming in an arbitrary way)
It is a practical question to ask about existence questions that require an axiom (axiom of choice often) which gives no constructive way of doing something. To what extent is that actually useful? And if we had an approximate construction of something without the axiom, but the thing itself may or may not exist depending on the axiom chosen, what would that mean?
It is also interesting to think about doing away with infinity and thinking about how annoying it would be to not be able to talk about pi or e or even sqrt(2). There would be no irrational numbers, presumably, in this framework.
One can then see the wonders that embracing infinity can lead to as a crutch for dealing with the horrendously messy world of the finite and discrete.
Discussing these issues when learning mathematics strikes me as elevating a great deal of bland rule absorption and gives students power over the tools of mathematics. So this debate is useful in my opinion though it would be absurd to try and draw a conclusion.
One last note about his discussion of the lack of rigor in QFT and that it was good. Recent work (at Rutgers[0]) suggests that at least some of the infinities and problems come from not appreciating the proper physical picture of what is actually happening. Nonetheless, the standard procedures worked to give us needed answers before such insight was discovered.
I think the basic answer should be that formal and exploratory mathematics are both useful and it is important to avoid dogma from either side.