Hacker News Reader

Comment by colechristensen | Hacker News Reader

Your package manager should be boring, extremely backward and forward compatible, and never broken. Experience has shown this not to be true for python. Several times over the years i’ve found myself, pinning, upgrading, downgrading, or otherwise juggling versions of setuptools and pip in order to work around some bug. Historically I have had far more problems with the machinery to install python packages I have had with all of the other python packages being installed combined, and that is absurd.

View on HN

I don't disagree with your claim, but like all moral claims, I don't see how it's actually useful or actionable. Children should not get cancer. People should not drive without seatbelts. Our optic nerve exit the back of our eye instead of having to go through blind spot on our retina. And yet here we are.

The Python ecosystem is older than our modern conception of how a package manager should behave. Any new package manager must build on top of that chaotic history. Absent a time machine, your comment just seems like a vague lament about the state of the world.

Actionable: use backward-compatible behavior by default. Use opt-in for new features. Question: is the new resolver opt-in or is it enabled by default in the new pip version?

It is enabled by default. I think there was a beta period where it was an option.

And there is an argument to disable the check.

How long should someone wait before deprecating the old behavior?

The action is simple, stop making breaking changes. There is frustration with breaking changes being in the history, but that is much less troubling than the reoccuring answer to the question “What is wrong with my code?” being “A new version of the package manager was released.”

It is also very possible to write code to completely work around old issues so the newest version of your package manager just works or at least outputs clear error messages with mitigation steps, but instead what you get is obscure stack traces when you run pip install.

Many of the breaking changes had to do with removing accidental features from the constraint file implementation. It seems a lot better to break that now rather than wait for people to accidentally start using and depending on those unintentional implementation quirks.

I tried the new version of pip and it's giving clear error messages when there are incompatibilities between versions. That should make your life easier.

This seems like an odd comment to make on a release which is making improvements, work done with the full knowledge that there’s room for improvement.

It’s also not universal: it’s been a decade since I’ve had to worry at all about that and even before then it was uncommon except on projects which weren’t trying to be very clever. If you had a clean setup this really hasn’t been the rule for a long time.

As long as you don't have lots of C or Cpp dependencies, Python packaging is apparently totally fine.

Unfortunately I don't live in that world :(

I mean, I do use a lot of C code and the things which break are external C level dependencies. You can blame Python for those but it won’t help fix them.

Well when pip installs an incompatible version of Numpy relative to my current setup (without warning me) then I do indeed blame pip.

To be fair, this release makes that much better, which I am extremely grateful for.

If a library doesn’t specify version compatibility and you don’t pin it but do choose to run an upgrade, why is that the installer’s problem? It’s like complaining about Toyota when you get a parking ticket.

See, this isn't necessary in other languages. All I want is an error, so that I can assess what the right thing to do is.

Honestly, I sometimes feel that people have stockholm syndrome with python packaging.

Nonetheless, I'm really happy about this release, as it will help to prevent these problems.

It's also worth noting that I didn't even learn Python for a long time because I kept breaking my install while trying to get started, so this stuff has real costs in terms of adoption.

The same problems happen in those languages — the underlying problem is someone not specifying version constraints they actually have. If you get more experience with those other languages you'll definitely encounter that with projects which aren't carefully following semantic versioning. In particular, this goes back to your previous confusion about C/C++ — if you're having a problem with those toolchains, the underlying problem is that they don't have a package manager at all and every downstream tool has to compensate for that.

The one part which is worse in Python than some of those other languages is that the default tool doesn't track all of the dependencies when you install a new package. If you're not using something like pipenv, poetry, etc. you can add "foo" to your requirements file and have it work now but break in the future if you aren't using a tool to pin specific versions (e.g. pip-tools). Some other tools like NPM or Cargo avoid that by default but that's certainly not true of all languages and it's something which is widely acknowledged as an area for improvement by the people recommending or working on the aforementioned tools.

So, my major comparator for ease of installation is a statistical programming language called R.

Both Python and R have equal numbers of C/C++ dependencies, but the only breakage I experience is with Python.

This is for a number of reasons:

1) R has CRAN (like CPAN, but for R). If your package does not build on the latest version of R, it becomes unavailable.

2) if your package has C++ dependencies, the installation will error out and say which headers its missing. This means that I can search for the header and install it.

3) Because of the removal of obsolete packages, dependencies just work and you can be assured that a package will build with a given version of R. Updates can be a problem, but they are opt-in at the package level.

The core issue with Python is the acceptance that different projects can have entirely different dependencies, which means that this breakage is relatively common.

Like, I build C/C++ on a semi-regular basis, and it can be super-painful, but Python is the only higher-level language that gives me this heartache.

Not to mention that if you install pip from the Linux repositories, you're setting yourself up for a world of pain (this actually stopped me from learning Python for a number of years).

To be fair, I mostly use conda for C/C++ dependencies as it works much better, but lots of packages are still only available through pip, which means I can't avoid the problem.

To reiterate, I am super happy with this release of pip, as it will make my life much better. It doesn't go far enough, but it is much, much better than what existed before.

Finally, Python packaging is a horrible wart on an otherwise good language, and I'm probably going to argue against using it in the future because of this.

> If your package does not build on the latest version of R, it becomes unavailable.

Wow. This would definitely make packaging much much easier, and it’s so cool R is able to do this.

I can’t imagine how angry people would get if Python ever goes there though. Each community has different tendencies, and one of Python’s I’ve observed is people really get pissed when you force them to upgrade anything.

Dependency management in C or C++ is always an absolute shit-show.

The only way to do it and to stay sane is to just include your dependencies in your source tree.

That is interesting. I haven't worried about setuptools since like 2009 and haven't cared about my pip version since like 2012.

I have had FAR more issues with performing Bundler/Ruby upgrades than I've ever had with Python.

I take it you're doing a lot of greenfield projects with the latest versions of Python then. I've encountered issues with Setuptools as recently as ~3 months ago:

https://github.com/pypa/setuptools/issues/2352

I've also had to play the pip version pinning game a few times in the past.

Here is one from September:

https://news.ycombinator.com/item?id=24336058

That one looks like an issue with the debian package trying to alter paths and blowing things up.

There's definitely recurring issues with distributions trying to handle python dependencies, which doesn't match how python handles dependencies.

Same here.

The entire Python ecosystem is like this. Different conflicting versions of the language, different package managers broken in different ways, etc. It’s so bad that the core user base of data scientists usually just use machines that have all their libraries pre installed.

Contrast this with a language like Rust or JS, where you run one command once to install everything

I don't know Rust well enough to comment, but doing a lot of development in both JS and Python, JS is no cakewalk either. I'd say I deal with at least as many packaging/versioning issues on JS as I do on Python.

Another issue is the two mutually incompatible ecosystems of async vs. non-async Python libraries and frameworks. You now have these entire mirror universes of code, with "[X]-async" versions of existing synchronous libraries, and new async frameworks which require using these libraries. The years of work that've gone into non-async libraries are now basically all for naught, with lots of people reinventing the wheel for every possible thing that interacts with a network. (I'm also not a fan of async/await in general.)

I'm a big Python fan; it was my first language and still by far my most-used and the one I know best. But the "developer experience" is undoubtedly a giant mess compared to something like Go or Rust. (Rust also appears to be having some growing pains related to async, but everything else seems way more solid.)

Despite Python's catchphrase "there should be one, and preferably only one, way to do it", Go really embodies this across every dimension. One way to install the runtime, one way to format, one way to package, one way to do concurrency, one way to test, one way to write most things. And every version is almost completely backwards compatible.

It's kind of crazy that a language that (justifiably) prides itself in its programming simplicity also makes you deal with tangled messes like https://xkcd.com/1987/

My one solace is that some all-in-one third-party tooling is finally close to the level of convenience you get with Rust. pyenv and poetry save a ton of headache when it comes to managing Python versions and packages, and "just work" in a pretty similar way as rustup and cargo.

> The years of work that've gone into non-async libraries are now basically all for naught,

That's fud. Non-async code lives in, is maintained, and is not affected by the async alternatives poping up.

We had libraries based on twisted for over a decade, which were basically in the same position, but you're not mentioning those?

> It's kind of crazy that a language that (justifiably) prides itself in its programming simplicity also makes you deal with tangled messes

That comic is satire with many paths which don't really exist. Reality is much simpler is you're explicit about which version you're using and use a single source rather than OS, homebrew, Anaconda and upstream release at the same time. With one version manager and virtual environment per project everything works almost exactly like in Rust.

You're right, "all for naught" was going too far and I apologize for the hyperbole, but it's kind of like a blockchain fork: most new projects are gradually moving to the new async side of things, and this new mirror universe is being built at an accelerating pace while the previous fork (synchronous libraries) is decelerating.

In 5, 10, 15 years, a lot of code that might have otherwise still had some utility may no longer be practical. A lot of such code might have become obsolete after that long anyway, but some of it might be used far less than it could've been solely because it's outside of the future async status quo.

There are some awesome Perl projects out there, and their work definitely wasn't for naught, but it's hard to justify using them for anything serious in 2020, kind of like how it might end up being hard to justify using synchronous Python in 2025 or 2030.

(Not in the sense of running a standalone application, in which case who cares what it's written with, but in terms of something that may require some kind of continuous development or interfacing with other code.)

>We had libraries based on twisted for over a decade, which were basically in the same position, but you're not mentioning those?

My understanding is Twisted doesn't suffer from the so-called "what color is your function problem": http://journal.stuffwithstuff.com/2015/02/01/what-color-is-y...

With Twisted, you need to swap out certain network calls with Twisted's own APIs to use its full feature set, but it has an easy way to defer existing synchronous/blocking code: https://twistedmatrix.com/documents/12.2.0/core/howto/gendef... There's no way to do something like this with async/await, besides some hacky third-party attempts, as far as I know.

But, yes, I'm not a huge fan of Twisted's model, either. I was thinking more along the lines of a green thread model like gevent, which, although controversial, doesn't require you to change a single line of code beyond a monkey patch line at the top of the file. I'm not really swayed by Twisted's creator's anti-green thread manifesto (https://glyph.twistedmatrix.com/2014/02/unyielding.html).

And any language with a baked-in green thread model like Go also avoids this problem. Synchronous code can be run asynchronously, or synchronously, with no red/blue barrier.

>That comic is satire with many paths which don't really exist.

Yes, of course it's hyperbole, but it's not that far off, in my opinion. Now that I use pyenv for everything I no longer run into any issues like these, but before, it could sometimes become a nightmare.

I love tools like pyenv and poetry and they make Python way easier to deal with (it's still my favorite language, for sure), but part of the point of this thread is that it would be nice if universal tools like these were baked into the actual release like how Rust's and Go's tools are baked into theirs.

Tons of people still aren't aware that things like pyenv or poetry or Anaconda exist, or that they should consider trying them. Or they want to try them but are dissuaded due to work policy about changing things or using new third-party tools, or even any kind of third-party tools. Someone else in this thread brought this up here: https://news.ycombinator.com/item?id=25255536

> My understanding is Twisted doesn't suffer from the so-called "what color is your function problem"

It really does. The link you mentioned shows how to delegate some sync work to a thread to make it play nicely, but you wouldn't want to do it in each request for example. Twisted is its own world and either you're in or you're out. The async/await markers are much more generic in this case.

> There's no way to do something like this with async/await, besides some hacky third-party attempts, as far as I know.

Spawn a thread, share a queue, send the sync call you want to do, await on a queue read. It's effectively the same thing twisted is doing there.

The JS ecosystem has just as many "issues" if you look at it objectively. Don't mean to be funny or lazy, but one need only look at this SO answer to see how weird/fragmented the JS package system is (the answer has a Table of Contents).

https://stackoverflow.com/a/39825582/1140382

That's not the JS package ecosystem. Those are all various techniques from 5 years ago for packaging frontend code to save download bandwidth.

In other words, your package manager should be perfect and bug free 5 years ago.

There are zero package managers from any environment for which these requirements are met.

I’m fairly certain that yum and apt meet these requirements for “traditional” package managers, I know Maven does as well.

I think it depends a lot upon the culture of the ecosystem, maven being fairly conservative is a natural consequence of Java being very enterprise focused.

That's easy to disprove. See the bugtrackers for those projects. They're not bug free. See why "aptitude" was introduced to cover "apt"s failures. Also one of the reasons we had the yum to dnf migration is to replace the worse resolver in yum with a sat-solver in dnf.

but python has been around for a while, I feel the frustration comes from the fact that other platforms seem to have managed to get a single working tool much earlier.

The only platforms that have excellent package management systems are the ones you haven’t used yet.

Npm is probably the closest. They had $10.6 million of investment to get it right.

It's mind boggling to me that such a popular, mature, and beloved language has such a terrible package manager. Pip is seemingly broken more times than not. Deploying anything is a nightmare unless you use a virtual environment, and even when you do you still gotta cross your fingers and hope pip doesn't decide to break at some point in the future.

Maybe I am weird, but I've had far more problems with anaconda and its package management than I ever have with pip. Conda routinely takes hours to resolve whatever unholy issues it's running into sometimes

installing code, packages is the hardest part of python programming, one of the reason I avoid it.

In my experience, python dependency management + library installations + code sharing is way easier than the following:

C#, C++ (Conan) and Java, Linux(apt, yum, etc) and most certainly JS/TS.

What is so hard about "installing code, packages" in python for you?

Java - all you have to do is mvn install clean build -> Maven is amazing at dependency management. Same in JS - npm install and npm start works like a charm.

And python all you have to do: pip install -r requirements.txt

Or even better: pip install -e /path/to/my/project

Works like a charm.

what about version management when conflict happens in dependencies that are installed vs needed. Admit Python is a big mess. That is why people recommend installing within another weird software called conda environment vs directly.

Corner case, and a tad overblown from an "academic" perspective if you ask me - I've been doing day job (and after-hours hobby with ML/CV) development in python for the last 15 years and had that issue maybe 2-3 times.

On the other hand, with my very limited exposure to my team's JS/TS FE components, I've almost each time encountered issues with the supposedly easy "npm install" command. Broken NPM installs, cryptic error messages (ok get this one on Python too tbf), lingering version lock files, global vs local package issues, broken nodejs installs, node_modules being committed, node_modules being un-deletable, etc. Anecdotally, I'd say my experience on Python packaging and management has been worlds-better than for other languages.

Python has it's issues sure, but from my point of view, packaging is not a big one in comparison to other langs.

"Experience has shown"?

Did you go create a test case? Or at least link to a specific issue?

Have you tried PHP? It has an amazing package manager called 'composer'. The current release is popular and super amazing (runs 1/2 the internet). Python is a terrible divided (2 or 3?) and gross language from 1991.

I remember composer as the package manager that required a gigabyte of RAM to install a few dozen of dependencies for some framework.

It can definitely be more than 1 now. New version should help, but also breaks.

Source, personal experience with my projects.

Insert "Poorly Hacked Perl" joke here ?? /s ohh - lets do vi vs emacs next! :-)

heh. the down votes are so pefect.

Reply on news.ycombinator.com