This is entirely predictable - Ruby does not have a big scientific computing community which happened to depend on every implementation detail of the hosting interpreter.
It is quite common to see "Python" libraries that are just thin bindings layers, they could just as well be "Tcl" libraries for that matter.
First of all, it contains a ton of custom C code, which to some extent could be extracted to a separate library in theory, but isn't. Second, a lot of that custom code interacts deeply with the Python C API, which historically was very open-ended. Even getting it to work on another implementation of Python was a challenge that took a long time to reach baseline usability.
You could forego Numpy and call out to a library like Eigen, but even then you have a huge amount of work ahead to achieve anything resembling feature parity.
Still, your lengthy explanation only confirms how much C and how little Python, that specific case happens to be.
Python workloads, with deep pocketed backers, do spend more time inside GPU or C runtime.