Hacker News Reader

Comment by ZeroCool2u | Hacker News Reader

Ruby is nice, but if you were to actually solve this problem for Python you'd print money. The deluge of knowledge required to actually navigate Python dependency management across all major OS's for the most complex use cases (typically ML stuff with deep compiler/tool chain deps) is nothing short of a nightmare. Especially on Windows, the rage I feel towards MSVC knows no bounds. venv, pip, (mini)conda, mamba, poetry, apt, yum... Obviously the last two are good to know anyways, but ugh the hours I've lost and will never get back to the rest.

Anyways, god speed my dudes.

View on HN

(founder here). The interaction between high level language package managers and system binaries or compiled extensions is definitely on our roadmap. For example the ruby net-ssh gem relies on specific versions of openssl being compiled on the machine. Do you think this is particularly difficult in ML?

Is your frustration mostly from needing compile all this non-python stuff across environments?

Yeah, you're right on the money. This is basically insane in ML, but _especially_ in corporate environments, think F500, where sudo is disabled in containers and experimentation (in terms of trying different package versions) is quite challenging due to security restrictions.

Frankly, it's one of the reasons that we're seeing a lot of excitement around stuff written in Rust, but with Python bindings[1][2]. Cargo makes it so easy to build across various platforms and then just produce statically compiled platform specific python wheels. If PyTorch and TF had been written using Rust, any other other ultra high perf language with a tool-chain as reliable as cargo, a lot of folks lives would be easier.

[1]: https://www.getdaft.io/

[2]: https://www.pola.rs/

Hmm, the Debian package of ruby-net-ssh doesn't even depend on OpenSSL at all.

Interesting. Should it? I don't have much experience installing ruby gems with the OS package managers, I've always done it through the language-specific ones like bundler / gem. Here's a github issue showing the kind of things that come up when the versions are mismatched: https://github.com/net-ssh/net-ssh/issues/843

I was probably looking at an older or newer version to you, likely it switched from OpenSSL to pure-Ruby implementations of the crypto stuff.

I don't think you'd print money just by telling people if a dependency is hard or easy to install though. You'd need to actually solve the problem of getting the dependency installed or suggesting some alternative solution that gets the code written.

E.g. if I'm banging my head against a wall trying to upgrade to the newest version of PyTorch for some graph embeddings feature and everything breaks, knowing it's high risk doesn't help me. I still need to get the thing installed so I can use the feature.

Okay yeah, totally agree. I already know it's hard to install. If they gave me a path they were confident would work in terms of deps/versions to upgrade to, that I would pay for if it worked well.

Do you work at a company forcing you to develop on Windows? You couldn't pay to do to that.

Reply on news.ycombinator.com