Speaking as former physicist, can only agree with you.

However, regarding the Dirac delta function an other generalized functions, even though they were introduced by physicists (I think Heaviside was an early proponent) in a hand wavy fashion, they were later put in rigorous mathematical footing in the theory of distributions. Distributions are nowadays used by mathematicians without any hesitation.

This also not the first instance of something like this, and it won't be the last. Physicists have come up with ad hoc methods that work, but they can't justify why with rigor. Some time later mathematicians formalize it and it becomes part of their tool set.