Comment by PaulHoule | Hacker News Reader

Comment by PaulHoule | Hacker News Reader

A few causes:

1. The python 2/python 3 split

2. Quite a few python lines such as 3.5 and the early 3.6 have deadly bugs

3. Diamond dependencies; That is package A depends on package B1 and package C depends on B2 but you can't have B1 and B2.

4. Packages installed in a user's home directory will normally be visible in any virtualenv or conda environment so if you install something there you are hosed.

5. The inverse not-invented-here syndrome which is expecting pip, conda, pipenv, poetry, pyenv, etc. to solve the problem for you as opposed to understanding what factors are at play.

The number of factors at play are finite and the problem is solvable but you have to start with xenophobia and distrust of the platform, get complete control of part of the space first and expand the space you control as opposed to attempt to find the 20% of the whole that gets you 80% of the way there because that last 20% is a woozy and will shift underneath you and make you doubt your sanity...

I feel like I've made it out the other side but I don't know if I could ever communicate what I've learned to a neurotypical.

What about Anaconda? Don’t they test things run together, I imagine updates might be a bit slower though

I pity the neurotypicals

Anaconda is part of the problem, not part of the solution.

Here are a few reasons:

1. Continuum analytics doesn't have a real business model and someday it will go away.

2. In the bad old days, before wheels, there really was a problem integrating C libraries with Python. Wheels work basically the same way conda packages do, so there is no problem. You can get an "official intel MKL" numpy, the only trouble is that (officially) the python packaging system has no way of saying "these N packages all satisfy the same dependency and you can install the one that is best for you".

3. "Testing together" is a wild goose chase because the combinatorical explosion of different libraries you could test together is practically unlimited.

4. Anaconda for a long time has been unable to package a "tensorflow-with-gpu-that-just-works" because NVIDIA won't let them. Without that, the window is broken and Anaconda is not giving you a problem free experience.

5. Numerous things about the way Anaconda is implemented make it slower than it has to be by a lot.

6. Since Anaconda doesn't confront all of the problems I've mentioned above you will still need to implement stabilizers to stabilize it, and if you're going to do that you can just build on wheels and have fewer things to understand, have go wrong, etc.

Can you expand on point 4? Conda-forge releases perfectly fine tensorflow-gpu builds, with the caveat that they don’t ship stubs or the actual NVIDIA driver with them so it’s not truly standalone, but the same can be said of pytorch or really any GPU-enabled package.

That's exactly what I mean.

A particular build of Tensorflow X requires version A of the CUDA library, version B of the CDNN library, etc.

It is a common situation if you work on a data science team or want to play with models you find on Github that some of them require X1, A1, B1 and some others require X2, A2, B2.

The CUDA and cuDNN libraries are ordinary userspace libraries so if you package them for anaconda you can install them into a virtualenv and have different versions of the libraries sitting side by side and never get an error because the library versions don't match -- and I've done that on both Windows and Linux.

Anaconda can't ship conda packages like the ones I describe because NVIDIA insists that you download the libraries from their website, register to get senseless spam, screw around with installers, etc.

So you have to be Senior Python Dev to address every issue right away without spending a night on a basic simple module. Why isn't Python more dev-friendly in our days, it's like it stuck in the 90s. I know building something without knowing what's under the hood is not the perfect example but sometimes I need swift solutions for a script that will basically run once and for all.

Reply on news.ycombinator.com