Just use the system packages! Fedora, Debian, AUR, brew/macports on macOS, etc are all a thing, use them.
It can take several months of work to approve one single package to the official repos, for a single distribution. And each have different rules and setup.
Now explain to me how you think this is going to work.
Also, do you place to force everyone to use chroot or containers to replace their virtual env systems to have variations on deps? Or maybe everybody should use nix?
Do you want to do that also for JS, Ruby and PHP?
Not all PyPi projects require C code.
> It can take several months of work to approve one single package to the official repos
This is a massive exaggeration. For Fedora it takes a couple of days, all of it being necessary review of the code and licenses. And yes, you do have to do that work, it's done by the distros themselves too.
> Also, do you place to force everyone to use chroot or containers to replace their virtual env systems to have variations on deps? Or maybe everybody should use nix?
Nix is a good fit for this, but so is Guix and probably Spack. With a sane implementation of PEP-725 (which I mention and describe in another comment on this post), users could freely choose whatever package manager they like to supply non-Python deps.
* Windows exists.
* The system package manager is meant for the system environment. The typical Python environment is separate from that - a container or a venv. The system Python installation is there to make the OS work, not to provide a working dev environment, and it's often not suitable as such (especially on Debian).
* There simply are far too many packages out there to make it workable.
* Part of the point of how Pip works is that compiled code can be pre-built for specific platforms. A system package manager typically loses that advantage and needs to rebuild from source.
* Installing Python packages "from source" commonly involves running arbitrary Python code at install time, by design. (The build system simply doesn't provide any standardized way to invoke compilers, nor to apply conditional logic - you're stuck with Turing-complete, unsandboxed Python. Python is really hard to sandbox anyway.) For obvious reasons it's not appropriate to do that as root, which rather interferes with using a system package manager. Yes, obviously packages that mirror PyPI distributions and are made to be installed with the system package manager, will be adapted to avoid security issues. But this requires significant, parallel maintenance effort. The devs of the individual projects likely don't have resources to spare to help distro maintainers, either.
If your projects can get away with having the same exact set of libraries installed, then no - just reuse the same virtual environment.
In between is, I would argue, mostly Pip's fault. There's no facility for installing via symlinks, or hard links, or `.pth` files, even though all of those would work fine in Python. Granted, using `.pth` files causes `sys.path` to blow up at startup, which some people claim is a performance issue (it's searched linearly) - but it's honestly probably not a meaningful problem in remotely normal cases (especially considering that module imports are cached in `sys.modules`).
Pip also unpacks wheels, even though Python could theoretically import from them directly (they're renamed zip files). This is partly for performance reasons but largely because it allows installed packages to read their own data files using relative paths and the ordinary file I/O API. For packages that don't need to do that, the egg format implemented a `zip_safe` flag, but for whatever reason, wheels don't have this.
I dream of the day where we only have one package manager on a single machine.
This is a slide backwards.
PEP 725 is trying to be exactly that (https://peps.python.org/pep-0725/). It just took this long, since the implementation of the new `pyproject.toml` system, to get around to it.
But aside from that, the system package manager is only appropriate for "dependencies" that need it - i.e. things that need to actually be part of the local system, not packages side-by-side in a virtual environment somewhere.
>Also it's sad how setup tools bdist_rpm seems to be considered deprecated.
Historically, Setuptools tried to be a complete solution for everything except actually fetching code. People were even expected to setup.py install locally after downloading something from GitHub or whatever. Nowadays it's supposed to be just a build backend, for the kind of building processes that can be automated under Pip's direction. `bdist_rpm` simply doesn't fit that mold. You're supposed to use a third-party build system instead.