It's not quite that simple. The core packages wrap fortran or C code but then additional packages are built on that. Going by the github breakdowns Pandas, SymPy or Jupyter are pretty much pure Python, matplotlib is 90% Python 10% C++ (though the Python ratio is probably skewed up by the in-repo examples).
And that high-performance Fortran or C code is at the root of so much scientific computing is an issue in Go considering the overhead of cgo calls.