Depends where in math, in things like particle physics things get all wibbly wobbly is my cat dead or alive. In things like engineering quite often what you're dealing with is probability based, but you just stack the deck so far in your favor the probability is 1.
As they say, building a bridge that doesn't fall down is easy. Building a bridge that barely doesn't fall down is much harder.
I have seen molecular biologists (jokingly) shake the voodoo "molecular biology maracas" over the PCR machine to try and replicate their result.
It's a human thing.
Surely Feynman made jested comments before running experiments. I'm sure some digging in his wonderful books and letters will find many examples.
In fact, just finished listening to a talk where a experimentalist was talking about how to get the fabrication yields of superconducting qubits from currently low double digit to 99.99+.
Biology is messy at a macro level is all I'm saying. I don't need a hundred people butting in saying "butt aschully phsyix and code is also messy and harder at a quantum level." I know. We know.