The problem is not so much infinities in mathematics (after all mathematics is just a game with any rules you care to create), but if mathematics with infinities should be used when modelling the real world.
I am certainly happy to be corrected, but I don’t think we have found any event in nature that requires infinities to explain it.