Hacker News Reader

Comment by lexicality | Hacker News Reader

I do a lot of maintenance work and every time I've encountered a (complex) project set up in this way that's older than ~6 months, it's bitrotted from breaking changes in the dependencies.

The python ecosystem does not stand still and seems quite happy to introduce breaking changes in non-major versions.

To my mind, there are 3 ways to make sure your python project of today will work in 2 years time:

  1. Have no dependencies 
  2. Destroy your virtual environment and reinstall it every day for the next 2 years and fix anything that breaks
  3. Freeze your entire dependency tree where it stands and occasionally do large breaking change updates

All 3 options suck but #3 is the least worst

View on HN

IMHO the problem with python package management isn't python package management. The biggest problem I've hit is when python depends on non-python - `pip install` failing because I need to `apt install libsomething-dev` is the big one, but also managing multiple versions of python itself (I count this as a dependency on non-python because the python3 binary isn't written in python - if it was, then pip+virtualenv could manage it like everything else). And also as you note there's just ecosystem churn.

If we could reboot all of computer science, I'd want one unified cross platform package manager with the same market takeover level as Git, and all languages designed to be aware of it.

No file system overlay stacking or anything, no file system hierarchy of different folders for different types of resources, just python style search paths and virtual environments, and no global environment allowed, ever.

But Python development is pretty nice at the moment so I can't complain.

Is option 3 exactly what you’re supposed to do? Freezing your dependency graph and/or explicitly denoting what version of the dependency you want are your best bets for avoiding problems like this

It's certainly what I'd go for and what I recommend to everyone, but the GP disagrees:

> I never mention transitive dependencies in my requirements.txt file, just direct dependencies and rely on pip to install all transitive libs.

> You dont even have to freeze the version, just list the name and pull up latest version whenever you run pip upgrade

3 is exactly what you're supposed to do.

A lot of people will assume that specifying major version upper bounds on dependencies is what you're supposed to do, but I've seen this fail more often than freezing dependencies.

The problem with major version upper bounds is that if it's possible to write a test case for a bug, it's possible to depend on broken behavior. Changing behavior in a way that breaks users should be a major version bump, but that's not actually how people use semver and semver isn't really described that way either. It's described in a way that makes people think that changes in type signatures are the predominant impetus to bump major versions.

I mention this ceiling pinning footgun in the article. It's an enormous pain in the ass to explain to folks, and some software engineers I've met are totally incredulous that that's "not the right thing to do"

Poetry makes it 10x worse with its `^` operator

Definitely! And the article you link to in this section is great too

It is, but Python software tends towards large dependency graphs which will quickly accumulate CVEs, and so this strategy greatly upsets your security/audit people.

(Obviously patching vulnerabilities is good and proper, but automatically flagged CVEs are only ever _potential_ at best; in many contexts most of them are not actual vulnerabilities.)

Yes. But then any transitive dependency stays there forever even when not needed anymore.

Initial requirements (only first level, version ranges) and dependency resolution results (all transitive packages, exact versions or hashes) are two very different things and should be treated separately.

You can implement and maintain it by hand with two requirements.txt files but it's rarely done this way. And really at this level you're better off with a normal package manager.

I work in JS and the issues are the same.

It's why NPM creates lockfiles that can freeze the version of direct and indirect deps.

fwiw, we built an oss plugin for poetry on option #3 when publishing for repeatable installs to automate translation of published wheel metadata from lock file, also supports mono repos.

https://github.com/cloud-custodian/poetry-plugin-freeze

What sucks is when you imply that your project will be unmaintained for 2 years, without any kind of update (not even basic security upgrades). This is a not a project, this is abandonware from day 1.

There was a time when you wrote a software for a version of the OS and it was done (except security fixes).

Obviously my profession gives me a ton of bias but in general it seems that for most people the squeaky wheel gets the grease and once a project is working it is ignored until it stops working again because there are only so many hours in the day and management has an infinite list of demands...

And then it breaks 2 years later and I'm called in because the person who wrote it left and no one else knows how it works because they didn't write any documentation etc etc etc

It's not like that simple 50 line Python script taking stuff from API A and stuffing them to API B is a full ass project with a JIRA project ID and a Product Manager.

It works, people forget it exists.

Then something breaks and I need to start fixing it. And the amount of work it takes to just update the packages is more than it'd take me to rewrite the whole thing in a language with a sane distribution mechanism (Go).

Reply on news.ycombinator.com