Gonum is probably what you are looking for, it should have most of the optimized linear algebra stuff you need.

https://github.com/gonum/gonum

For sparse, I think the best is still Intel's sparse libraries for MKL on CPU, it's just not a workload that runs well on GPU compared to dense SGEMM. You can call it via CGo.

For GPU offloading, there should be a decent amount of CUDA bindings available for Go right now, but I'm not as familiar in that front since I'm building my own GPU compute runtime in Vulkan, but it's still very much an experimental work in progress right now.