Hacker News Reader

Comment by Waterluvian | Hacker News Reader

The release of this pretty much hosed my morning at work. Suddenly CI wouldn’t build because this broke things in pyyaml and awscli.

And then I got to re-learn how very bad Poetry and python dependency management is. I’m trying to update pyyaml to 6.0.1 and Poetry systematically downloaded every single major, minor, patch version of another dependency in trying to find which would fit the version constraints. Took half an hour.

View on HN

Why are you blindly upgrading major versions of your tools?

Sounds like fundamental design issue more than anything.

You'd hope people would learn this time, but they'll probably just keep blaming their tools...

Version dependencies aren’t propagated and can be outside your local control. As an example elsewhere in this thread: PyYAML did not have a pinned Cython dependency with a maximum version before this, so if your project depended on a pinned version of PyYAML it would break when trying to build it after Cython 3.0.0 released.

Why are you constantly rebuilding dependencies? Surely you should only rebuild if there is a new (and vetted) version

Lmao you’d think. But this is not the default or recommended pip/setuptools behavior: https://pip.pypa.io/en/stable/reference/build-system/pyproje...

The usual methods to control versions with a lockfile don’t work here because it’s a build dependency. You’d need to disable build isolation and install the right build tools in your environment before installing pyyaml but that can become a mess real quick.

It's a fundamental design issue with Python package management that makes it impossible to pin the version of build-time dependencies. You can do everything seemingly right, with pip-tools creating lockfiles containing hashes of downloaded packages, stuff seems to be stable for a year+. And then suddenly your build starts failing because the lockfiles only contain runtime-dependencies, not build-time-dependencies, and some library you are depending on just was silently downloading the latest version of some package you've never heard of, until it broke.

The only semi-sane workaround I know of is to use pip's `--only-binary` option to prevent any automatic builds at package installation time. Then you usually will also need your own package server for storing precompiled wheels for those third-party-dependencies that don't published compiled packages on pypi. This way you build the packages using some random tool version only once, so if it works the first time it will keep working.

In other language ecosystems were the package management isn't built on a tower of shit, this problem doesn't exist in the first place.

Doesn't poetry have a solution for this?

>you are depending on just was silently downloading the latest version of some package you've never heard of, until it broke.

How? Are you running `pip install` every time you run your pipeline? Why do you need to constantly be reinstalling your packages?

If you're using poetry, why not just add a `cython = "<3.0"` to your pyproject.toml file? You can lock down subdependency versions like that, and presumably the versions of everything that worked yesterday still work today.

It’s not part of my dependencies and it wasn’t on my radar. What I learned was that pyyaml had a new version that fixed the error being thrown by pyyaml.

If a top level dependency has build time dependencies then those do not respect the version locks of your pyproject.toml unless you disable build isolation.

> Poetry systematically downloaded every single major, minor, patch version of another dependency in trying to find which would fit the version constraints.

Great news! Better do that then have to do it manually

But yes it sucks, still Poetry is much better than Pip (where it just gives up and tell you to go deal with the issue yourself!)

Now why do people care about Cython and especially in things like dependencies I don't understand. awscli shoud be an independent package! "oh but it is faster" are you counting the time it takes to fix things when they break because you are depending on a fragile thing?

Other systems can do this with far less effort. It’s really a total joke.

Well the way npm solves this is having every package with its own copy of dependencies, maybe that's the way python should go, but I'm not sure if other package managers are much better than that (let's say Cargo in Rust)

And that's pretty much why you use binary package distributions like conda, despite all its flaws a sudden release of cython will never break your deps builds because there's nothing to build.

You can force pip to only use wheels. Then if something fails, build your own wheel in your own (pinned) environment. Could be done in multi-stage docker build, for example.

If one of the build deps has no wheel then you're fucked, though. At some level you have to rely on a toolchain.

I think it doesn’t have a choice in downloading all versions if they use a setup.py config file. You have to get the sources to run it and get the package’s metadata.

Reply on news.ycombinator.com