Pipenv: Python Dev Workflow for Humans | Hacker News Reader

Pipenv: Python Dev Workflow for Humans

pipenv.pypa.io

31 pointsBorlands63 comments

About a year ago or so I moved to pyenv and poetry. I have used python for over 10 years it was a game changer. I recommend this guide if your still using pip and venv.

https://cjolowicz.github.io/posts/hypermodern-python-01-setu...

is it not venv under the hood?

It uses venv under the hood. But a car is more than just its transmission.

Very much seconded. You don't really want to use Pipenv, Poetry works much better and AFAIK is more popular.

Does poetry still only work for libraries or does it work well for apps too now?

What do you mean? You mean for installing Python apps on your computer? I just use `pip install --user` for that.

No like if you want to put a webapp in a virtualenv in a docker to run somewhere.

I remember pipenv used to support this pretty well but it was out if scope for poetry.

Edit: there's a 'poetry run' command now so it might be fixed. Don't remember what was broken.

Is this not applicable to Windows? The article mentions Unix, Linux, and Mac. Sorry for the n00b question; I'm just an amateur Python data analyst.

I'm using poetry on Windows with no issues whatsoever.

Honest experience: I switched to this early on because at the time PyPA was pushing it really heavy, and Kenneth's past experience at Heroku meant it was added to their Python buildpacks far too soon.

At the time, it was unusable and community responses wildly swung between "its not official", its "PyPA endorsed", "don't like it build your own". Which are fine responses beta version of software, not when you convince the community and several companies that its a stable production ready, langauge endorsed package manager. Ultimately, pipenv in Heroku python buildpack was broken for a long time because of it.

Despite being assured it was stable, there were daily Pipenv releases at times, until they stopped for some reason for months.

In summary: avoid pipenv unless you really have to use it, otherwise - use poetry.

I had the same experience. I got an error message that seemed incorrect, and I opened an issue about it, only to get a sarcastic response:

https://github.com/pypa/pipenv/issues/1722

I switched to Poetry after that, as Poetry did meet my expectations.

We had many issues with pipenv and migrated all our projects to Poetry, which I would recommend over pipenv for solving this problem.

I like that pipenv also handles virtualenv creation, but I too moved my projects to Poetry after all of the various Pipenv-maintainer fiascos

Poetry also creates virtualenvs for you I believe

Yes it does

By default, poetry automatically creates virtualenvs for you. Though you can override that behavior in your local or global poetry config.

Oops. What I meant is, Pipenv manages python versions for you. You tell it what version of python your project uses, and pipenv will make sure to create the venv using that version of python. Great! Solves managing python versions.

With Poetry, the venv is created with whatever version of python that Poetry was installed with. So if you (like me) follow the setup guide on macOS, Poetry will always create python 2.7 virtualenvs until you uninstall it and re-install it under python 3.X. Annoying.

It will only use whatever version is currently activated if that's compatible with the python version requirement specified in pyproject.toml. If it isn't, then it will look for an installed python that is, and create the virtualenv using that instead.

I'm not sure I completely love that behavior; it would be nice if it would default toward trying to use the minimum supported version, even if the currently active one is compatible. But I guess that wouldn't do anything that tox couldn't do better.

You can also explicitly tell it what version to use for a project with a command like `poetry env 3.8`.

EW. Avoid. For the tenth time in these comments, use poetry.

It's a real shame PyPA endorsed this to be honest, I hope that can be reversed at some point in the future.

Poetry is basically perfect, it does what it says and has no surprises.

Well, except one thing: If I do `poetry update <somepackage>`, it starts upgrading other packages as well. I'm not sure what that's about...

As long as you install it correctly, which can be difficult on systems where python2 is the default.

Actually, I tried that the other day. It complains about having the wrong Python, but will auto-select a version compatible with your project and run fine. For me, it selected Python 3.7 because that's what the pyproject.toml specified, even though I was running it under Python 2.

Such systems (RHEL6, Ubuntu 16.04, mac OS 10.14) are rapidly approaching EOL or already in paid support only lifecycle stages

Poetry is great but sometimes I finding jarring that its error messages show up in the terminal with the full python exception as well.

What does this tool provide over an in-project .venv/ directory created by venv and populated by pip from an etc/pip/requirement.txt file containing all the packages by version?

There seems to be a lot of activity in this area for a seemingly solved problem. Having solved it, people appear to wish to expand the problem to something bigger (such as multiple interpreter versions, for example)

I believe the most-common case to be: 1 Choose python version and install on operating system., 2 create .venv/ directory. 3, install packages from package system (pip)., 4 freeze versions to local file. 5 use installed .venv/ and do work.

I've got some scripts I use for the above [1] -- must give it some TLC, and I don't understand why people furiously want more than this.

[1] https://github.com/gjvc/python-template-project

The consensus among non-Python developers is that that is not good enough.

The problems are:

requirements.txt is way too flexible, so users want a lock file to freeze stuff in place.

You can use pip freeze to generate a sort-of-lockfile but it's more manual than tools like Cargo/npm/bundler which do it automatically. And more adhoc as to what you call it. Combined with needing to mange venvs and sourcing etc. people want a standard script that just figures out its context from the now standard "explicit requirements"/lockfile setup.

It doesn't handle versioning Python itself, which is expected of it for some reason, even though nobody cares that npm/Cargo/bundler don't version their languages.

The lock file shortcoming is better remedied by pip-tools [1] as recommended by a neighboring comment.

[1]: https://github.com/jazzband/pip-tools

I felt the same way about only needing requirements.txt until I managed a project that needed to be compatible across many versions of python and some of it's dependencies were renamed from one version to another. I highly recommend you take a look at the hypermodern python guide https://cjolowicz.github.io/posts/hypermodern-python-01-setu...

Pipenv and poetry help, but, even with them, it's still not good enough. setup.py is a real challenge. Packages being able to execute arbitrary code at install time can make it very difficult to get a truly reproducible python build. Details of how pip (which pipenv and poetry still use) handles packages means that doing anything that even hints of a monorepo is a delicate subject on the best of days.

Agreed.

And the solution to the multiple interpreter versions problem should be orthogonal to venv like pyenv [1] instead of overlapping with venv like pipenv.

[1]: https://github.com/pyenv/pyenv

It is a solved problem for other languages. Pip has soooo many issues.

The biggest IMO is pip doesn't provide a way to pin indirect dependencies, so there is no good way to ensure that other people working on a project are also using the same dependency versions. You can do it with extra tooling of course - and that's what poetry and pipenv do.

Isn’t this what pip freeze is for?

pip freeze only pins the version of direct, not transitive dependencies. Meaning you don't get deterministic builds.

What do you mean? If you run pip freeze inside a venv it will print every user-installed module and their dependencies. When installing them in a new venv as a requirements file, you end up with the exact same set of modules.

Agree. Works fine for me exactly as you describe. I don't understand why people furiously want more than this for the common case. (Choose python version, create .venv/ directory, install packages, use installed .venv/ and do work.)

What happens when you use a different version of the same library that a dependency uses?

Sometimes it works, sometimes it doesn't. This is indeed a problem, but one for which there isn't an elegant solution.

When multiple versions of the same library are allowed to coexist in different dependency trees there is less incentive to solve these conflicts and versions proliferate (see npm).

I'd much rather solve the occasional conflict (usually solved either by settling on some older version of both parent dependencies - and later advancing as possible - or just trying to fix it and send a patch upstream).

Issues start occurring when you, for example, delete dependencies because with pip freeze you cannot ensure that you have deleted their dependencies also. The most common solution to this (that poetry and pipenv use) is to provide a lockfile to track transitive dependencies and their versions, without that (except for manually curating your dependencies) you can't ensure that you get a reproducible environment.

OK, but it does manage the versions of transitive dependencies, and there's nothing in that process stopping deterministic builds.

Adding/removing top level dependencies over time does require the use of two files (the top level requirements and the frozen/locked requirements which lists everything). Or you can list the top level requirements in setup.py and let requirements.txt be the lockfile. It would be nice if pip managed this lockfile automatically, but I'm not really interested in adding any of these newer tools to my toolchain just to manage a lockfile.

There are many packaging and distribution frustrations in Python, I don't think pip's management of dependency lists is one of them

Not so. I use pip freeze to manage my dependencies, including all transitive dependencies, and my builds are deterministic.

How do you include checksums in your freeze to catch when the package changes on the pypi server using only pip?

I don’t. PyPI no longer allows reuploading the same release.

Even worse-- there's no simple way for pip to print out the full dependency tree (including transitive dependencies). This makes it a nightmare when it complains about some transitive dependency version conflict and you have no idea what's causing it to be pulled in

I just so wish that Python tooling would be better, in large part pip is at fault here. If there was no conda I probably wouldn't even touch Python as things look very bleak especially if you need to run anything across different operating systems.

I thought `pip freeze > requirements.txt` would give you a list of all dependecies, even indirect ones. Am I missing something?

aka UNIX maxim of tools which do one (and only one) thing, and do it well

Have had a lot of really bad experiences with pipenv. Would recommend pip-tools instead.

Yes, pipenv is so frustratingly slow I found it practically unusable. poetry is also a good alternative.

We had a medium size app and it took literally 30 minutes on a MacBook Pro to resolve (not install, just resolve) dependencies. This was a few years ago so perhaps performance improved, but it was abysmal and made it impossible to meet our goals of fast CI/CD.

Seeing all the comments in this thread that people have had bad experiences with pipenv, this contrasts with my own experience which has been pretty good. How has pipenv failed for everyone?

Gonna take a look at poetry, but would love to hear what problems people have had with pipenv?

1. Dependency resolution algorithm failed in places that Poetry handles fine.

2. When Airflow added a dependency choice (based on licences) the setup script required an ENV var choosing one to be set, pipenv swallowed the stdout output and dumped a Pip stacktrace that didn't help diagnose the issue.

3. Mysterious bugs on new pipenv releases that usually manifested as a Pip stack-trace or setuptools stack-trace.

4. Sometimes the bugs were installed Pip version dependent, which made replication hard.

There was other stuff, but can't quite remember off the top of my head.

But yeah, we spent a lot of time trying to figure out why pipenv had broken again. Poetry has been a breeze in comparison.

There is also Pdm now I recently learned, which is supposed to be a more modern alternative. As an alternative to poetry that is; pipenv is yesterday’s solution.

https://pdm.fming.dev/

Ah, that uses PEP-582-style package directories, that's interesting! Though I don't know if we need an alternative to Poetry, I'd rather we just standardized on one tool at this point.

Recommend poetry over pipenv

There wasn't a full release for a year and it broke my pip on Ubuntu 18.04 as I recall - literally it stopped pip from working after I installed it (can't remember why). Avoid.

I had also the terrible misfortune of trying it on Ubuntu, when dependencies were screwed there were no way to fix it even manually, it was like adding insult to the injury that is pip itself.

Poetry all the way here, been a blessing! The fact that Pyenv cannot solve some environments that Poetry can is a no go.

Worth noting that Sébastien Eustace is an author of the provisional PEP 621: "Storing project metadata in pyproject.toml"

https://www.python.org/dev/peps/pep-0621/

Pipenv is problematic in a number of ways, succinctly summarised in this article: https://chriswarrick.com/blog/2018/07/17/pipenv-promises-a-l...

Any insights on why not use conda instead?

Conda's a good replacement for the whole pip/pyenv/venv/stuff-that-isnt-technically-python-that-gets-hacked-in-with-setup.py mess, but, if you're publishing packages, pipenv and poetry go a step further to help with building those as well.

How's this better than pipenv? Also, does it work on Heroku?

> How's this better than pipenv?

it's not, because it is literally pipenv

> Also, does it work on Heroku?

yes

Ah my mistake, I should read more. Thank you!

Reply on news.ycombinator.com