> I might be misremembering, but I think fastmath was one of the flags explicitly warned against in the Gentoo manual.

It is, here: https://wiki.gentoo.org/wiki/GCC_optimization#But_I_get_bett...