Well shit.

I run some large ML models in my home PC and I get NaN's and some out of range floats every month or so. I have spent hours debugging but doing the same computation with the same random seeds does not recreate the problem.

How about GPU's and their GDDR SDRAM? Do they have parity bits?