Hacker News Reader

Comment by ninkendo | Hacker News Reader

I count at least a half dozen “just use X” replies to this comment, for at least a half dozen values of X, where X is some wrapper on top of pip or a replacement for pip or some virtual environment or some alternative to a virtual environment etc etc etc.

Why is python dependency management so cancerously bad? Why are there so many “solutions” to this problem that seem to be out of date as soon as they exist?

Are python engineers just bad, or?

(Background: I never used python except for one time when I took a coursera ML course and was immediately assaulted with conda/miniconda/venv/pip/etc etc and immediately came away with a terrible impression of the ecosystem.)

View on HN

I think it is worth separating the Python ML ecosystem from the rest. While traditional Python environment management has many sore points, it is usually not terrible (though there are many gotchas still-to-this-day-problems that should have been corrected long ago).

The ML system is a whole another stack of problems. The elephant in the room is Nvidia who is not known for playing well with others. Aside from that, the state of the art in ML is churning rapidly as new improvements are identified.

Two problems intersect:

- You can’t have two versions of the same package in the namespace at the same time. - The Python ecosystem is very bad at backwards compatibility

This means that you might require one package that requires foo below version 1.2 and another package that requires foo version 2 and above.

There is no good solution to the above problem.

This problem is amplified when lots of the packages were written by academics 10 years ago and are no longer maintained.

The bad solutions are: 1) Have 2 venvs - not always possible and if you keep making venvs you’ll have loads of them. 2) Rewrite your code to only use one library 3) Update one of the libraries 4) Don’t care about the mismatch and cross your fingers that the old one will work with the newer library.

Most of the tooling follows approach 1 or 4

Disk space is cheap, so where it's possible to have 2 (or more) venvs, that seems easiest. The problem with venv is that they don't automatically activate. I've been using a very simple wrapper around python to automatically activate venvs so I can just cd into the directory and do python foo.py and have it use the local venv.

I threw it online at https://github.com/fragmede/python-wool/

You’re already managing a few hundred dependencies and their versions. Each venv roughly doubles the number of dependencies and they all have slightly different versions.

Now your 15 venvs deep, and have over 3000 different package version combinations installed. Your job is to upgrade them right now because of a level 8 CVE

Yeah that sucks. something like:

    for venv in $(find ~/projects/ -type d -name 'venv'); do
      (
        source ${venv}/bin/activate
        pip install --upgrade pip
        pip install --upgrade package_with_cve
        deactivate
      ) &
    done

should do the trick. Sucks that we're in that world, but let's not work any harder than we have to.

I forgot about parallel, which would do better than spawning off all the pip install --upgrade at once.

After many years of Django development I've settled on what I find the simplest solution. It includes activation of virtual environments when I cd into a directory.

pyenv for installing and managing python versions.

direnv for managing environments and environment variables (a highly underrated package imo).

With those two installed I just include a .envrc file in every project. It looks like this:

layout python ~/.pyenv/versions/3.11.0/bin/python3

export VARIABLE1=variable1

export VARIABLE2=variable2

etc...

Oh nice. Stick

    PYTHONPATH="`pwd`/venv/lib/python-3.11/site-packages"

or whatever in .envrc and Bob's your uncle.

It's not bad. It works really well. There's always room for improvement. That's technology for you. Python probably does attract more than it's fair share of bad engineers, though.

Just replace python with mac or apple in your comment and I think you will understand.

Unironically yes, it really is that bad. A moderately bad language that happened to have some popular data science and math libraries from the beginning.

I can only imagine it seemed like an oasis to R which is bottom tier.

So when you combine data scientists, academics, mathematicians, juniors, grifters selling courses…

things like bad package management, horrible developer experience, absolutely no drive in the ecosystem to improve anything beyond the “pinnacle” of wrapping C libraries are all both inevitable and symptoms of a poorly designed ecosystem.

Reply on news.ycombinator.com