My understanding is that's not correct. It's possible to do general purpose computing on a GPU with things like CUDA, but your code has to be specifically designed that way and is tied to the drivers/hardware you implement it for. It's not going to automatically happen if you write some matrix code.

I would guess most people aren't doing anything like that. If you are just running a basic site and database you don't need a GPU.