Hacker News Reader

Comment by woodruffw | Hacker News Reader

Who is "they"? PyPI is an index; it doesn't control your installing client.

(This is a larger issue - or feature, depending on your perspective - with Python packaging. But it's important to understand that PyPI itself can't force `pip` or any other client to pick any particular resolution order between indices.)

View on HN

For all intents and purposes "pip" is the official client. It is referenced in the official documentation https://docs.python.org/3/installing/index.html

The fact that pip is the official client isn’t in dispute. The point was that pip and PyPI are different entities, per a larger pattern of devolved ownership/control/standards-over-tools in Python packaging. PyPI has little to no say over how pip and other tools choose to handle resolutions across multiple indices.

The PSF has a saying in which is the default installer and how pypi is run.

PSF has little control over anything. The Python ecosystem is consensus-based.

They have a say insofar as they can participate in the same standards process as everyone else. But no, the PSF has no unique say in how PyPI is run, or how pip behaves. This is a pretty fundamental aspect of how Python-qua-ecosystem works.

They have a say in that if it doesn't behave like they want they can point the documentation to something that does. If pip is the tool linked in the documentation it's the official one that has the PSF's blessing, clearly.

We're going in circles. PSF can't unilaterally change any documentation of particular relevance here; the most immediately relevant docs would be controlled by PyPA and PyPI itself. The former has a standards/community review process, and the latter is particular to PyPI.

But again: this has nothing to do with blessings or not. The fact that pip is the official installer and PyPI is the official index does not mean that everything about them stems from an official edict. That's not how Python's community is structured, and it's certainly not how the technical development on anything in Python packaging has ever progressed.

Who controls the python.org domain that I linked? Am I wrong to think that PSF controls it?

PSF controls that domain. But that domain doesn’t host PyPI (anymore) or the PyPA docs, so I’m not sure what connection you’re making there.

https://docs.python.org/3/installing/index.html

so it's PSF's decision to document using pip, right? And they might decide to change that page without asking for anyone's consent right?

So they also don't get to complain about pip's shortcomings because it is their decision to point users towards pip. Correct?

Have I been explicit enough for you now?

You've pointed to a stub document that basically explains Python packaging and why pip comes with Python (which, notably, distributions are fond of breaking).

This doesn't somehow imply that Python has a top-down authority structure where the PSF dictates the development flow of PyPI, pip, or any other official/semi-official/blessed tooling or infrastructure. That's not how Python works as a community.

The PSF is also not complaining here about pip's shortcomings. I'm the only one here and I don't represent the PSF, nor am I complaining: I think pip is great. I'm trying to explain (apparently unsuccessfully) why pip (and PyPI's) behavior isn't always 100% congruous with a single stream of development thinking. As mentioned above there are more optimal structures given different kinds of governance and community, but this one aligns with the (IMO good) values that the Python community espouses.

I think the PSF has the means and opportunity to say "please don't use xxx, use yyy instead". Which is why conda and linux distributions are not seen on the same footing at all.

I think it's more because Conda has explicitly positioned itself for a specific domain (scientific Python) and because Linux distributions aren't operating at the same level of specificity. The success of uv has demonstrated that Python packaging tools don't need PSF or PyPA affiliation to be extremely popular; they just have to be good.

But this final claim is essentially right: PSF can suggest things, and the community will (to some extent) accept those suggestions as blessed. But this doesn't mean that the PSF can dictate what would essentially be a significant breaking change to pip's behavior.

> Who is "they"?

The PyPI and Pip developers of course.

Those are largely disjoint sets, and the post in question is about PyPI.

So? The issue requires coordination between Pip and PyPI. I don't see what point you're trying to make.

The issue does not require coordination; that's the point. It's a behavioral aspect of `pip` that's completely opaque to PyPI, because all PyPI does is serve index responses to installers. It doesn't know how many indices the installer contacts, or the order in which it contacts them (and it has no good reason to know those things, ever).

The simplest way to fix this problem is to support namespaces, which PyPI absolutely does need to be aware of.

This would not be meaningfully addressed by namespaces, since there's no authoritative, authenticated unique name system across indices. Two separate indices can (and will, based on what ecosystems like piwheels do[1]) advertise `foo/*`-namespaced packages, leaving installers where they are today.

(I think namespacing is a good idea regardless, if only because it eliminates artificial scarcity in a one-level namespace.)

[1]: https://www.piwheels.org/

This absolutely would be meaningfully addressed by namespaces because the typical use case is PyPI + a private repo you control. Register the namespace in both repos and you're done.

If you disagree I would love to hear of a concrete way that solution would be vulnerable.

> Register the namespace in both repos and you're done.

That’s the operative part of “authoritative.” It’s a distributed trust problem, and there’s no particular guarantee that your namespace on one index will be honored by another. Namespacing is great for eliminating scarcity on one index at a time; I don’t think it helps much with this kind of cross-index security.

Right, so it would provide a solution to the specific problem I'm taking about.

> since there's no authoritative, authenticated unique name system across indices

Domains provide a globally unique namespace and ownership can be verified automatically with domain validation. Bluesky did an ok job of it, but they didn't do anything to account for domain ownership changes and re-validation is non-existent, which is disappointing to see from the first big adopter since the oversight will eventually invite criticism.

I've wanted domain validated namespaces for 5+ years. Here's a comment I made about using domain validated namespaces in package managers a couple of years ago [1]:

---

I think one possible solution to that would be to assume namespaces can have their ownership changed and build something that works with that assumption.

Think along the lines of having 'pypi.org/example.com' be a redirect to an immutable organization; 'pypi.org/abcd1234'. If a new domain owner wants to take over the namespace they won't have access to the existing account and re-validating to take ownership would force them to use a different immutable organization; 'pypi.org/ef567890'.

If you have a package locking system (like NPM), it would lock to the immutable organization and any updates that resolve to a new organization could throw a warning and require explicit approval. If you think of it like an organization lock:

    v1:

        pypi.org/example.com --> pypi.org/abcd1234

    v2:

        pypi.org/example.com --> pypi.org/ef123456

If you go from v1 to v2 you know there was an ownership change or, at the very least, an event that you need to investigate.

Losing control of a domain would be recoverable because existing artifacts wouldn't be impacted and you could use the immutable organization to publish the change since that's technically the source of truth for the artifacts. Put another way, the immutable organization has a pointer back the current domain validated namespace:

    v1:

        pypi.org/abcd1234 --> example.com

    v2:

        pypi.org/abcd1234 --> example.net

If you go from v1 to v2 you know the owner of the artifacts you want has moved from the domain example.com to example.net. The package manager could give a warning about this and let an artifact consumer approve it, but it's less risky than the change above because the owner of 'abcd1234' hasn't changed and you're already trusting them.

---

1. https://news.ycombinator.com/item?id=32754029

Reply on news.ycombinator.com