Hacker News Reader

Comment by postpawl | Hacker News Reader

That probably works for smaller projects without many dependencies, but it’s just going to install the sub-dependency versions that satisfy whatever comes last in the requirements file. The pip docs describe that situation here: https://pip.pypa.io/en/latest/topics/dependency-resolution/

The pip docs also suggest using pip-tools to create lock files. Pip-tools is only for creating lock files (it’s not trying to fix virtualenvs like poetry is), and it works great.

View on HN

Yeah our codebase has a requirements file that takes over an hour to install with the new dependency resolver (and over 10 minutes using the deprecated resolver.) This is on a 6c/12t ryzen with 32g of ram and a gigabit connection.

This has been my experience, too, and with a moderately simple set of dependencies. I don't understand why this is considered acceptable enough to deprecate the old resolver.

it seems like you are posting on the wrong platform, github is that --> way :}

the pip team has introduced strict performance checks to ensure the new resolver was not significantly slower than the old. they solicited requirements files to weed out situations exactly like the one you describe. for example https://github.com/pypa/pip/issues/8664

if you want it improved, please report it.

I believe pipenv (not pyenv!) is also a viable option for correct versioning, though I'm not actually sure whether it or pip-tools is more actively developed these days. Last I used pipenv though (~2 years ago), it was a nicer virtualenv + pip-tools combination, but had worse version resolution / less useful verbose output for no apparent reason (since iirc it shares tons of code with pip-tools).

Putting that aside though: yes, 100% pip-tools or an equivalent (which pyenv is not). It's the only sane way to both freeze dependencies for reproducibility, and maintain upgradability. I've used pip-tools for years on many large, complex projects, and it has been a huge benefit every single time. And it routinely revealed significant library-incompatibility problems that teams had only luckily dodged due to not using feature X yet, because pip's resolver has been so braindead for forever.

Pipenv quickly becomes unusably slow. As in half hour to change one dependency. Poetry is strictly better, and I say that believing poetry is not a great solution either. There aren't many good, general recommendations to give people regarding Python package management, but "not pipenv" is one of the few I'm confident in.

Good to know, thanks :)

How long ago did you try pipenv? I don't use it for large projects (just a few dozens dependencies), it is not fast but it works (I expect it to be just a convenience wrapper around corresponding functionality provided by pip/pip-tools (there were substantial changes in pip's dependency resolution).

Interesting. So use `pip-compile` instead of `pip freeze > requirements.txt`?

Sort of - you need to make a requirements.in file that specifies only your top level dependencies, then you run pip-compile on that file to generate the requirements.txt.

So another requirements.in file ignoring PEP621? I'm confused...

Maybe I will just try first. pip-tools seems to acknowledging PEP621

https://pip-tools.readthedocs.io/en/latest/changelog/?highli...

https://github.com/jazzband/pip-tools/issues/1047

pip-tools predates PEP621 by ~5-8 years, depending on how you count. So yes, it "ignores" it.

Reply on news.ycombinator.com