It is amusing to note that on mini computers of the 1970's that what we now would call a "floating point co-processor" was a hardware option. Even with the Intel 8086, a "math" coprocessor was a separate 8087 chip.

On mini computers you used a different version of the language compiler depending on whether your hardware had this coprocessor or not. Typically you didn't make this choice, but the system administration people made sure the correct compilers were installed to generate hardware supported floating point or software floating point. That tradition seemed to make its way into PCs. It was a while before you could just assume that everyone had hardware supported floating point. Oh, and every language's software floating point was annoyingly different. So no portable way of writing binary floating point -- you had to convert to/from ASCII.

GPUs are like a new fangled hardware floating point external to the CPU. Now to gradually become part of the main system again.