Hacker News Reader

Comment by scrollaway | Hacker News Reader

> It's not suited for libraries and is not designed to be.

That's kind of the problem. Why on earth are libraries and apps getting a different treatment? The JS ecosystem manages to have one tool for apps and libraries. One too for installing and publishing. All of it with lockfile support, workspace/"virtualenv" support, etc. And somehow, it's not confusing.

Adding one more tool to the stack is a really funky step forward. Yes, it brings lockfiles. Cool, although we already kind of had those (--freeze). Packaging in Python is a mess and I'm more and more in the camp that as long as Pipenv keeps flouting itself as "the better solution", all the while not covering all the basic use cases, we've gone backwards and are in even more of a mess.

View on HN

And it isn't only Javascript that manages to have a single tool for this. Clojure has a single tool. Java has a single tool. Rust has a single tool.

This is a solved problem across a variety of popular and mainstream programming languages. I don't mean to suggest that the problem isn't complicated, but this isn't a problem that doesn't have a wealth of previously written solutions to look at for inspiration.

Right, exactly! It bugs me that we have a wealth of examples and could have just said: "Look. Here's yarn. Do that, but for Python. Don't try to be too creative."

Poetry looks like it did just that though and I'm warming up to it at a very high speed.

What's the "one tool" for Java? Is it Maven? Ivy? sbt? Gradle? You may say some of those are "build tools" rather than dependency tools, but I don't see how it's different than what we are discussing in Python.

My Java app declares its dependencies in a build.sbt file using Scala syntax and has them cached in an Ivy directory. Yours declares them in a pom file using XML syntax and has them cached in a Maven directory. Neither of us even tries to do lockfiles and instead just has the CI server build an "uber jar" with all the dependencies bundled in.

People use sbt for Java? Thats’s...surprising.

Most shops I know use one of maven or gradle. I can’t think of any other serious contenders in Java (if you still use ant: please consider alternatives).

We don't need lockfiles in Java land because we generally use version ranges very carefully and rely on package developers following semver - and we certainly don't use "just whatever the latest release is, dude" like shown in the example Pipfile: https://pipenv.readthedocs.io/en/latest/basics/#example-pipf... (the asterisks)

I get why Python needs lockfiles, but goddamn, that need is a symptom of the mess of managing Python dependencies.

There's still a long way to go - I use Airflow for ETL management, and I'm using pipenv to manage it - except Pipenv can't create a lockfile because a dependency of a dependency of Airflow requires in flask-wtf <= 0.7, while another dependency of a dependency requires ==0.8

In a Maven pom.xml I can easily exclude or override the conflicting dependency manually if needed, but I can't in a Pipfile.

rely on package developers following semver

Well, that seems naive. Almost 1/4th of Maven Central libraries broke binary compatibility in patch updates: https://avandeursen.com/2014/10/09/semantic-versioning-in-ma...

Netflix, at least, doesn't agree that Java doesn't need lockfiles: https://github.com/nebula-plugins/gradle-dependency-lock-plu...

With my very small sample size, maven and gradle's lack of version locking just means it gets tacked on in terrible ways when needed.

> We don't need lockfiles in Java land because we generally use version ranges very carefully

Yes, which is part of the reason why lots of Java projects almost never update their dependencies, because no one remembers why that version was chosen. Splitting what you want from what you have is important to communicate this.

> There's still a long way to go - I use Airflow for ETL management, and I'm using pipenv to manage it - except Pipenv can't create a lockfile because a dependency of a dependency of Airflow requires in flask-wtf <= 0.7, while another dependency of a dependency requires ==0.8

I'm not sure what you expect it would be able to do here...

Enforce Flask-wtf 0.8 and see if it works.

Clojure has two tools.. designed to be used as the single tool though.

Boot, and Lein. I'm partial to boot lately.

libraries and apps _should_ get different treatment. A library often has to coexist with other unknown code (open world assumption), an application you're controlling is effectively existing in one specific universe: the one you define (closed world assumption).

You wouldn't write your application to support 5 versions of Django, but you _probably_ would do so for a library.

That said, I do basically agree about `pip` existing already. We could have built a tool to manipulate `requirements.txt` files instead of introducing another format and a toolchain that is _much_ slower and brittler. Though ultimately at this point Python packaging woes feel like they are at a much lower level (the fact that libraries end up being "installed" mean that preparing all your dependencies to go out to multiple servers is a mess).

An internal application can exist in one specific universe. Most of the time.

The "library" workflow works for applications too. Put your direct dependencies in setup.py. Build wheels of everything and upload them to an internal PyPI server. Pin everything in requirements.txt.

> We could have built a tool to manipulate `requirements.txt` files

There's pip-compile from https://pypi.org/project/pip-tools/ that does exactly that. Pipenv uses its resolving mechanism if I'm not mistaken. It produces standard req.txt file with versions pinned and supports separate dev requirements. It had some bug with upgrades last time I checked though, not sure whether it's resolved, currently considering using it for projects at work.

Have used pip-tools for a long time... its very uncomplicated and just makes things pretty darn simple.

Since its all just requirements files with pip-tools, its been fairly commitment free, in that I haven't had to substantially rework any major bits of workflow to use it - nor would I to remove it. Not sure I could say the same thing about pipenv and/or poetry.

Pipenv keeps flouting itself

That would be weird. Perhaps you mean 'flaunting'?

Yes, typo :)

Reply on news.ycombinator.com