Python setuptools v50 breaks pip installation on Debian/Ubuntu

Slightly exaggeratting, but in the same way, you could say "the designers of the C language did nothing wrong with creating that much undefined behavior, any competent programmer just needs to know what they can do and what not".

I am not blaming the developers of setuptools or pip which have very limited resources. But the solution they created so far is just not good enough, and its technical properties contribute to these problems. And other language environments clearly do it better.

Another aspect is that the community accepts libraries which break backward compatibility. This will become an ever increasing problem as soon as it becomes more frequent that programs rely on different libraries which in turn happen to have indirect dependencies which require conflicting versions of the same thing. There is no way to resolve that in Python. It would be much better that libraries which break backward compatibility would get a different name altogether each time, because in this case you could include the different libraries in the same program.

But Python's library authors don't, and this is probably because they want people to keep using their library by given the mere impression that it is compatible, while it is not. And this is just not honest communication. If you break compatibility, you should tell the users of your library. Or better even you don't break it in the first place.

Similarly, most software used to tell you in the README file to do a "make && make install", sometimes installing into /usr/local but not always. If a user followed these instructions and overwrote half of their distro-provided /usr/lib, things would break in the same way for the same reasons.

The solution is to not do that. Don't mix distro-provided files with things installed from other sources. (As for the problem of people providing bad instructions, I don't know the solution. Perhaps remind novices that all instructions come with an implicit context and may not apply fully to their specific case?)

  # In your CI pipeline, something like this:
  $ cat Dockerfile  
  ARG release_tag
  FROM baseimg:${release_tag}
  # etc; i think the container metadata keeps the base version, or you can store it in the container with a RUN line
  
  $ base_version_tag=$(./get-latest-versioned-tag.sh some-docker-repo:my_prefix/my_image:stable)
  $ new_container_version=$(./make-new-version.sh)
  $ docker build -t my_img:${new_container_version} --build-arg release_tag=${base_version_tag}

If you do this with a specific stable tree (like debian:8-stable or something) you should only get security updates, but if you use :latest, that's a total nightmare.

https://metacpan.org/pod/Carton

Instead, I recommend just running the other executables in the bin directory. `./bin/python` will drop you into a Python REPL with all the packages installed in that virtualenv available. Your IDE should let you specify what Python interpreter to use for running tests and other integrations; give it the path to that Python executable and everything should Just Work. Likewise, if your package uses setuptools entrypoints, they'll be installed into the bin directory too, and you can just run them with no further configuration.

For what it's worth, Rust (for one) works pretty much the same way. The only difference is that Rust doesn't have a concept of dependencies installed system-wide, they're always handled per-project like venvs. As a result, they don't have decades of stale tutorials online telling people to do the wrong thing.

From VSCode I just use the integrated terminal to create the venv, vscode's default python extensions recognizes in-process venvs by default and uses them. (The way it installs tooling is broken for them, though, but that's a bug in the extension.)

Though it would be nice if Code integrated poetry so that just having a poetry-compatible pyproject.toml was sufficient.

> Working with .NET, Rust, Node, go, Haskell, etc... I have no such issues.

For most of those (Haskell with Cabal included), they have the same basic issue and solve it with a project file which is pretty similar in purpose and effect to poetry’s use of pyproject.toml.

Node definitely has this issue. There's a reason Node NVM[1] exists.

[1] https://github.com/nvm-sh/nvm

Respective solutions: NuGet, cargo, npm, go get, cabal. They all solve pretty much the same issue as venv and prepare your project in pretty much the same way. Raw venv can be a bit rough, but poetry solves that.

  M-x pyvenv-activate

  (pyvenv-activate "~/code/python/venv_dir/emacs_venv")

  C-P
  Python: select interpreter

From a stability, reliability, and CI/CD performance standpoint, I would discourage building your own Python. Things break more the more often your base version changes.

Basically, start with "pip (without isolation of projects)" until it doesn't work for you because of library version clashes, then move to either pipenv or "Python -m venv + ./bin/activate + pip" (which are basically the same thing).

Poetry isn't as widely used.