I happen to know that pytorch is a pain to maintain in packaging systems. It has a complex build system, many non-python dependencies, and massive build times. I don't know how this factors into the dissatisfaction with a C++ implementation, but I wouldn't be surprised if it were a factor.

In other words, python binary wheels are harder to maintain than source-only python packages. And pytorch uses more than a few. I can't imagine Julia makes the problem much simpler. The main pain point is probably the lack of standard, multi-environment packaging solutions for natively compiled code.

I don't know what it would take for this sort of pain point to improve significantly. Some standards around how C, C++, and Fortran projects are packaged would help. This would allow projects to build on top of existing natively compiled tech a lot better. Maybe the biggest reason those languages don't have the same "ecosystem" as python is utter lack of packaging standardization.