Give it in a hands of bad programmer, you are fucked harder than any pornstar I have ever and yet to seen in my 21 year lifespan.
You are right about pip too, I don't know how did they achieve to come with a package manager that is worse than NPM. Such a feat.
Maven is a correct package manager. Maven is able to get a global view of the metadata for all of the packages that conform to the specification so it can find a correct solution for dependencies before it downloads any JARs.
Historically, Python used egg files that required you to run the setup.py (and download dev dependencies) before you knew what the dependencies of a package are. This has the real advantage of being able to change your dependency list based on your environment (say loading a GPU-enabled or platform-specific version of a dependency) but "doing the right thing" would involve doing all this work in isolated environments and doing an awful lot of work.
Thus you get the "optimistic" approach used by pip which is to start installing packages one at a time and hope for the best. There is no correct strategy to recover when it paints itself into a corner.
Almost all pypi packages are wheels now and they just introduced the ability to download just the metadata file (a few years back when I was struggling with Python packaging at a startup I did find you could accomplish the same with two or three http range requests since you can find the superblock and directory information at the end of the wheel and then fetching the metadata is easy.) If a package is only available as an egg you can do what anaconda does which is make a wheel of it for your specific configuration and stash it in your private repository.
We might see a correct system emerge out of this, but...
Python has suffered from people who think an 80% or 90% correct solution is acceptable but that's just wrong. Until ChatGPT came along nobody thought an 80 or 90% sorting algorithm is acceptable. Certainly people could decide they need or don't need certain features (for instance pyenv) but there are certain things that are either wrong or right and you can either invest a finite amount of effort into getting things right or struggle forever pushing bubbles around under the rug.
Python also needs a good answer for environmental variation. For instance there is scikit-learn and there is a version of scikit-learn compiled against Intel's MKL library. The latter package should be tagged as a substitute for the first so that somebody who wants to use it can swap it out without making any changes to the package configuration file (can't do that because you don't want to check those changes into version control.) There are some affordances for packages having optional features but these just aren't adequate to the task.