Python's dominance is holding us back. We need a stack with a more principled approach to environments and native dependencies.
Since then the whole python ecosystem has gotten worse.
We are building towers on quicksand.
It's not about python, it's about people who don't care about dependencies.
13 years ago when I was trying to explore the field R seemed to be the most popular, but looks like not anymore. (I didn't get into the field, and do just a regular SWE, so I'm not aware of the trends).
There is also a lot of development in Elixir ecosystem around the subject [1].
[1](https://dashbit.co/blog/elixir-ml-s1-2024-mlir-arrow-instruc...)
There are Microsoft-backed F# bindings for Spark and Torch, but no one seems interested. And this is despite a pretty compelling proposition of lightweight syntax, strong typing, script-ability and great performance.
The answer will probably be JavaScript.
Everyone already knows the language - all that's missing is operator overloading and a few key bindings.
For exactly the reasons your mentioned, i feel like F# would have been the perfect match for both MLE/ETL(spark pipeline) work and some of the deep learning/graph modelisation such as pytorch. Saddly, even from MSFT, the investment in F# as dried up