Hacker News Reader

Comment by user5994461 | Hacker News Reader

Python is unified on a requirements.txt file with pip, to manage dependencies.

Not "a" requirements.txt file, sadly. I've seen packages with different requirements for building, installing, and testing. All with their own damned txt file. Oh! And don't forget the duplication in setup.py -- you'll want to put your requirements there, too.

As far as I know, Pip doesn't have the notion of dev dependencies, so you need at least two requirements[.dev].txt files.

> As far as I know, Python doesn't have the notion of dev dependencies

Python doesn't have the notion of dependencies.

Pip doesn't have dev dependencies, poetry does.

That's (the deps, not the dev deps) likely to change with pip 20.3! All hail the dependency resolver.

It's already available behind a feature flag.

https://pythoninsider.blogspot.com/2020/07/upgrade-pip-20-2-...

D’oh! Thanks. :)

I don’t think this is true. I’ve personally seen the following (and I’m positive there are others):

1. setup.py with self contained dependencies (this is my general preference)

2. setup.py loading from requirements.txt

3. Standalone requirements.txt

4. requirements.txt generated from requirements.in (or similar)

5. pyproject.toml + Poetry with a dependencies section

requirements.txt has a lot of deficiencies for package management (compare to the various features Yarn or Cargo offer). The biggest weakness is transitive dependency locking: even if you specify the exact version of the dependencies you want, Pip can still resolve to a different version of their dependencies, and even then there’s no verification that the content matches what is expected beyond the version number (other tools use a content checksum). This can cause a long tail of reproducibility and maintenance issues.

It’s also a tiny subset of what the parent describes.

It also can't separate dev vs test vs prod dependencies, for example. Maintaining separate files for those is not exactly a convenient solution

I strongly recommend https://github.com/jazzband/pip-tools to solve this. It provides a simple script to take a requirements file and "compile" a full specification of all transitive dependencies. You check both files into the repo, point pip at the generated file, and manually modify the other one. It means you often don't need to pin requirements manually at all, and the versions will be explicitly updated whenever you choose to recompile your requirements.

When I first set up my requirements.txt's, I usually set up a venv, install the things that I need and then run "pip freeze" to get a list of all deps including transitive deps, and put them all in the requirements.txts.

I do sometimes feel that people are making rube goldberg machines out of their package management in an attempt to avoid just writing down all their deps.

>The biggest weakness is transitive dependency locking: even if you specify the exact version of the dependencies you want, Pip can still resolve to a different version of their dependencies

Isn't that what pip freeze is for?

The trouble there is that if you freeze you can't unfreeze. A project needs the equivalent of both gemfile and gemfile.lock (unless you're using a system like maven that got things right the first time), but there's no standard way of having both of them in Python.

This is a big oversimplification. There are often multiple setup.txt files for different environments, virtualenv setup scripts, tox files, several different build config files, manifest files, and so on. Python probably has more magic config files than C#, which is quite an accomplishment.

But there are usually several versions of python with several pips, possibly in combination with anaconda with multiple environments, a combination of named and rolling releases, and a SAT solver that's always slow but occasionally spontaneously decides to become extra slow and extend your build time by hours.

Oh and I forgot about setup.py. That too.

I never have a unified requirements file.

I tried to downvote this, but I think it’s already downvoted to the max. There is so much wrong with this pattern.

Please use something like Poetry to manage deps.

Reply on news.ycombinator.com