If crates wants a policy that stores crates forever then just store crates forever. What I'm talking about just affects how new crates and new versions of existing crates get "uploaded".
Imagine you call your crate foo.com/bar. In order to push your crate the crates server needs to verify that you actually own the foo.com/bar URL.
There are many ways to do that. A simple one is to just have the crates server pull code from that URL instead of pushing it. If you want to push (e.g. to work around reachability issues such as firewalls) it's just the matter of devising one or more ways to prove that you "own" that handle. For example by serving some challenge response in a html meta tag
All of the existing services it would make sense to delegate to are more mutable than we wanted for our system.
What I wanted to highlight was:
1. I like a package repository that has "namespaces". In fact that's what I like about Go approach to package repository from day zero, despite other rough edges.
2. Classic approach to "namespacing" like maven's is susceptible to name squatting or just "split authority": who can claim the "org.apache" org? First comes, first served? If that doesn't match who owns the apache.org "brand" on the wider internet, it's not technically a problem of the package repository but surely it's a practical one.
3. Name squatting problems exist for DNS too; it's a know problem, with an existing administration and dispute resolution body, a marketplace and whatnot. If you cannot get hold of foo.com you'll settle on getfoo.com or (some abomination like that) and that will be your brand as far as online presence is concerned. It's only natural for artifacts you want to associate to that owner to just piggy back on that more prominent identity rather than have to fight it independently for getfoo.com and perhaps land on gotfoo.com because getfoo.com has been taken by somebody else.
4. Now, if you decided that you want namespacing and given the observed consequences of allowing your users to compete for namespace names independently of their main online presence, I claim there is a technical approach that can achieve your stated goal of reproducible builds. The goproxy is almost what you want. The main different is that the go team aims at a best effort service and thus want to call it a caching mirror. They could have chosen to offer a more strict retention guarantee. Go packages are indeed immutable. The sumdb transparent log is there to enforce that invariant. It's really just about somebody willing to take the burden of storing the package bits forever, which is what I understood you choose to do with crates.
(I do remember maven before that check was implemented and it was chaotic)
I'd like to understand more about the arguments for and against a URL-based module naming / resolution approach. Could you point me to any past discussions on this for further reading?
I am not 100% sure exactly what you're asking for, so if you could say a teeny bit more than that, that would help me understand what part of the space you're trying to explore. It would also be handy to know what problem you are trying to solve, there are several here.
I think the way Go and ES Modules (Deno / browsers) represent module naming and resolution by URLs has some nice benefits:
* decentralized module hosting
* could be extended beyond HTTP / Git (ie. import modules via "ipfs://")
* re-uses existing web infrastructure for namespace ownership (ie. via DNS)
* possibility of hosting backend and frontend modules for multiple languages in a single HTTP registry
* modules may become unresolvable from their original URL (link rot)
* cannot guarantee immutability at the original URL
* semver support becomes a concern of the registry, and some registries may not support it (ie. the logic to resolve "https://example.com/my-module/1.x.x/lib.rs")
* more difficult to discover (browse and search) all available modules
* proxies and dep lockfiles of file hashes to help prevent link rot and immutability
* possibility for open source registry implementations with built-in semver support
* central index of modules which can be searched / browsed, if modules are submitted to it
I'd be very interested in learning more about the costs / benefits of the various approaches and the Rust team's discussions on modules. I really appreciate the team's thorough approach to design decisions, as it's manifested into a really cleanly-designed language.
Thanks!
1. the only part of this that's been in recent discussions is using the URL as a namespace. This has led to questions around how those are represented in code, given that URLs are not identifiers. Also, using DNS for namespaces is mutable and costs money, this is a huge barrier to adoption that we did not want. Publishing should be free and easy.
2. I think you've already identified some of the tradeoffs that made this be rejected: for reproducible build reasons, referring to external services is considered unacceptable. This is due to stuff like downtime of services we don't control, but also the mutability of those services. In theory, you could proxy them, but it's not as clear what this actually buys you, because to get those advantages back, you'd want to always refer to the proxy, and now you've re-centralized everything.
Another way to look at this is that all of the solutions you've proposed are significantly more complicated than what we've built. That complexity is for good reasons, but given 2, it's not clear that they're good enough to justify the costs. The current situation is significantly simpler for all parties, and the end experience ends up the same anyway.
I hope that helps!