Generally floating point works fine for most things. Every now and then it doesn't, and you have to be aware enough of its weaknesses to (a) detect this, (b) understand what the hell is going on, and (c) find a workaround for it.
Turned out that in certain cases, the factors were all tiny and so both the numerator and denominator became denormalized and the division returned a NaN, ruining all further computations.
After a lot of debugging trying to find the source of the NaNs and understanding why exactly this happened the solution was quite clear, simply expand it back out: (a/b) * (c/d) * (e/f).
This was because due to the nature of the math being implemented, those ratios would always be roughly around order unity, even though each individual variable could potentially contain a very small number.
This costs us like 0.1% performance, but made the code handle all inputs without issue.
I think this would have been a more relevant link
https://en.wikipedia.org/wiki/LogSumExp#log-sum-exp_trick_fo...